Project Details
Projekt
The Status of Linguistic Constraints in Neural Language Models
Applicant
Professor Dr. Erhard W. Hinrichs
Subject Area
Applied Linguistics, Computational Linguistics
Term
since 2026
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 579372518
We want to investigate the extent to which generative language models (LMs) are capable of acquiring abstract linguistic knowledge beyond the factual information presented in the training data. By abstract linguistic knowledge, we mean knowledge about linguistic mechanisms and patterns acquired by LMs without being explicitly trained for it. Recent linguistic studies suggest that LMs exhibit such abstraction capabilities over linguistic rules. Given this promising state of the art, we want to investigate whether such abstraction capabilities also extend to sets of interacting linguistic constraints in phonology and morphology that can be in conflict with one another and that thus require constraint conflict resolution. We want to explore to what extent current transformer models can already capture such constraint interactions. Specifically, we want to address the following research questions: 1) Are transformer-based generative LMs able to abstract linguistic constraints from the training data? 2) If abstraction capabilities are confirmed, how similar are the abstracted constraints to the frameworks established in the linguistic literature? 3) If abstraction capabilities are not confirmed, how does explicit insertion of constraints in the prompt change the model generations? By addressing these questions, we strive to make novel and innovative contributions to the research questions summarized under the rubrics LM capabilities, Ontological Status, and Explanatory Potential of the Priority Programme. We carefully chose three linguistic phenomena such that the complexity of the constraint space governing these phenomena is suitably diverse. All three phenomena can be formalized as a sequence-to-sequence task of generating the most probable output string given an input sequence. This makes them ideally suited for an LM analysis. Investigating the constraint interaction for each of these phenomena with LMs can provide a cue about the abstraction capabilities of LMs. Our main hypothesis for this study is: transformer-based generative LMs do abstract and generalize linguistic constraints from the training data. We furthermore predict a negative correlation between the complexity of the constraint space for a phenomenon, and the extent to which LMs abstract these constraints. To evaluate our hypotheses, we will adopt a set of methods from mechanistic interpretability studies. Specifically, we will be 1) producing possible output variants, resulting from different constraint violations on the input, and 2) inspecting the probabilities assigned to them by LMs, in different settings. This will allow us to mitigate the interference of factual knowledge and concentrate on abstract linguistic knowledge acquired by LMs.
DFG Programme
Priority Programmes
