Project Details
Projekt
Systemic Robustness Assessments of Language Models for Cross-Linguistic Research using Formally Related Structures (FORESTS)
Subject Area
Applied Linguistics, Computational Linguistics
General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
Term
since 2026
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 579277669
The central objective of this project is to develop a novel interdisciplinary approach that leverages language models (=LMs) as tools for cross-linguistic research and linguistic theories as tools for systemically assessing LMs' robustness. For this goal, we operationalize linguistic theories to assess how robust an LM’s “holistic” syntactic knowledge is, by moving from evaluation on single phenomena to systemic assessments of networks of formally-related structures (=FORESTs). FORESTs are a network of abstract structures that share underlying syntactic properties within languages and/or across languages. For example `Who does Peter like _ best?' and `What do you think that Mary bought _?' share the dependency of a filler `Who/What' to a gap (_) but differ with respect to the presence of embedding. We use such networks of frequent and grammatical filler-gap dependencies and compare them to infrequent and ungrammatical island-configurations as well as infrequent but grammatical parasitic gap constructions like `Who did you kiss _ without knowing _?', where an illicit gap in an island becomes well-formed due to a gap outside the island. Based on theoretically informed sets of FORESTs, we develop systemic assessment procedures that test for the presence of ``holistic'' syntactic knowledge in an LM. We further develop robustness scoring of these assessments for families of models that, in the next step, allow to test predictions of different theoretical analyses of parasitic gaps. Current theoretical analyses of parasitic gap structures make different predictions as to which other structures are close members in a network of FORESTs. We use these differences in theories to compare results of acceptability judgments of parasitic gaps and related structures in humans with LMs' performance on these structures, by manipulating training data input to include different forests. We will first set up this procedure for a set of theoretically well-described FORESTs and languages. Our main goal is to connect LMs' assessments and cutting-edge cross-linguistic research, focusing on the theoretically challenging case of parasitic gaps. Bringing together theoretical linguistic knowledge and computational expertise in LMs, the project addresses the research questions of the Priority Programme LaSTing in various ways. First, the project contributes to robust assessment by designing benchmark materials in a more theory-driven and generalizable way, including a cross-linguistic perspective. Second, experiments that vary input, model size, and architecture will lead to a better understanding of the limits of syntax learning in LMs and their transferability to other languages. In the long run, these insights can contribute to making LMs more resource-efficient and sustainable. Finally, the project aims to conduct research on foundational questions regarding the explanatory power of LMs for linguistic theory building.
DFG Programme
Priority Programmes
