SYNTH: Evaluating Transformer Models to Generate Synthetic Replications, Nomological Nets, and Peer Reviews

• Applicants: Dr. Björn Hommel; Dr. Ruben Lennartz
• Subject Area: Personality Psychology, Clinical and Medical Psychology, Methodology
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 546323839

The scientific reform movement has identified and implemented methods to improve the robustness, reliability, and replicability of research. However, these methods are labor-intensive and partaking in them is not consistently rewarded. For example, researchers are encouraged to preregister (i.e., declare design, hypotheses, and analysis plan before data collection), and peer reviewers are asked to check that deviations from these protocols are transparently reported. This, however, is quite time-consuming, and reviewers usually do not read preregistrations when they evaluate a full manuscript. This threatens the promise of preregistration. Recent advances in the field of natural language processing have yielded large language models (LLMs), which hold tremendous promise to reduce the costs of tasks that the scientific community has neglected. SYNTH aims to increase research reliability by developing tools to support researchers in three core domains: (1) Synthetic Replications uses a fine-tuned LLM to predict the pattern that survey participants will show when responding to questionnaires (e.g., personality tests). Using this synthetic data will allow reviewers to screen for potential issues of replicability when examining newly developed questionnaires prior to collecting empirical data. As part of the program, a vast selection of existing instruments will be synthetically replicated and empirically re-examined, if the LLM identifies possible concerns about consistency in results. (2) The Synthetic Nomological Net is an LLM-based semantic search engine, which maps thousands of instruments used in the behavioral sciences. As many published instruments may measure the same concept, the engine can be used to identify potential redundancies. Likewise, the Synthetic Nomological Net serves as a tool for researchers, enabling them to proactively verify the existence of an existing measure in the field, thereby preventing unnecessary duplication of effort in creating new ones. (3) The Synthetic Peer is an LLM specially trained to scrutinize and compare preregistration protocols with final research papers, highlighting deviations from the original protocol to aid peer-reviewers in their work. In a collaborative effort with the journal Psychological Science, SYNTH implements a one-year field study to assess the effectiveness of the Synthetic Peer. Alongside the development and evaluation of these novel applications for LLMs in research, SYNTH will also show how putting these applications in the hands of the research community can foster careful risk management and put their use on firm ethical foundations. The program highlights the importance of consent, privacy, fairness, and transparency and uses LLMs exclusively to prioritize, decisions are always made by a human being (concerns often termed ethical artificial intelligence). SYNTH meticulously monitors fairness, potential biases and its impact on the scientific landscape as a whole.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2317: META-REP: A Meta-scientific Programme to Analyse and Optimise Replicability in the Behavioural, Social, and Cognitive Sciences
• International Connection: Australia, Switzerland
• Partner Organisation: Schweizerischer Nationalfonds (SNF)
• Cooperation Partners: Professor Dr. Malte Elson; Dr. Juri Opitz; Professorin Dr. Simine Vazire