Project Details
Projekt
Addressing the generalizability crisis with large language models
Applicant
Dr. Dirk Wulff
Subject Area
General, Cognitive and Mathematical Psychology
Term
since 2024
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 546419617
The field of psychology has been grappling with a crisis of generalizability, where empirical findings are specific to a task or fail to replicate across contexts. In turn, traditional solutions that emphasize increased methodological stringency do not, per se, address the challenge of taming a sprawling array of constructs and measurements whose relations are largely unknown. A reevaluation of the relation between constructs and measures is imperative, alongside a structured approach to anticipate the circumstances under which generalization across measures or contexts will or will not materialize. This proposal suggests a language-based approach to this problem that promises to help estimate the nomological network connecting psychological constructs and associated measures. Language, integral to both theoretical formulation and data collection methods in psychology (e.g., self-reports, experimental instructions), can now be leveraged with the help of advanced language models, presenting a unique opportunity for both conceptual and empirical clarification. Specifically, this proposal suggests using language embeddings from state-of-the-art language models for an extensive examination of the relations between constructs and associated self-report and behavioral measures. This strategy promises to clarify the conceptual overlap between theoretical constructs and their operationalizations, as well as to help forecast the convergent and divergent validity of extant and novel psychological measures. The anticipated outcome is a clarified nomological network of constructs and associated measures. Ultimately, this research seeks to address the issue of limited generalizability in psychology, enhancing our understanding of the conditions under which empirical phenomena can be expected to replicate across measures and contexts.
DFG Programme
Priority Programmes
Cooperation Partner
Dr. Marcel Binz
