SocEnRep - Automating Reproducibility in Economics and the Social Sciences

• Applicants: Professor Dr. Jörg Ankel-Peters; Dr. Arnim Bleier; Professor Dr. Johannes Breuer; Dr. Sebastian Kranz; Dr. Alexander Rieber; Professor Dr. Ansgar Scherp
• Subject Area: Empirical Social Research ; Methods in Artificial Intelligence and Machine Learning ; Statistics and Econometrics
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 551687338

Reproducibility is an important cornerstone for making scientific research transparent and credible. Empirical investigations have repeatedly revealed issues related to the reproducibility of research in economics and the social sciences. Key factors that negatively affect reproducibility include that it is time-consuming and not sufficiently acknowledged and rewarded by the scientific community. This relates both to ensuring that one’s research is reproducible as well as reproducibility checks during and after publication (by journal/data editors, reviewers, or other researchers). The "SocEnRep - Automating Reproducibility in Economics and the Social Sciences" project addresses these challenges by developing an e-research toolchain to streamline systematic information extraction and reproducibility checks for authors, reviewers and editors, as well as for replicators and researchers who perform meta-studies. The toolchain automates the mapping of information from research articles (PDF or HTML) and their associated replication packages (source code and study data) with a focus on regression tables and code in Stata and R. The project has three main objectives: Objective 1 “Automate reproducibility checks”: We will develop and refine a toolchain that automatically extracts and executes Stata and R code from replication packages and compares the output with tables in corresponding articles. These automatic checks will produce detailed reports highlighting inconsistencies between an article and its code supplement. This can substantially reduce the workload for the authors writing a manuscript and increase the effectiveness of data editors, reviewers, and replicators seeking to verify results. Objective 2 “Support meta-studies”: We will build a searchable database of extracted information from articles and replication packages (including regression specifications, detailed regression results, and methodological choices). This will facilitate meta-studies on the robustness of empirical evidence and methodological trends across a large body of research. Objective 3 “Researcher training”: To promote and facilitate the use of the SocEnRep toolchain developed in the project, we will create comprehensive training materials and organize workshops. These activities aim to enhance researchers' reproducibility practices and will particularly target early-career researchers to enable them to use our toolchain in everyday research practice. The ultimate goal of the SocEnRep e-research toolchain, training materials, and additional output (such as publications and the reproducibility database) is to transform the research culture in economics and the social sciences towards making reproducibility the norm. By developing and testing solutions for facilitating and promoting reproducibility, the project aims to enhance the rigor, quality, and trustworthiness of research in economics and the social sciences.

• DFG Programme: Research data and software (Scientific Library Services and Information Systems)