Project Details
Projekt
Replicability and Robustness of Instrumental Variables and Randomized Controlled Trials in Economics
Applicant
Professor Dr. Jörg Ankel-Peters
Subject Area
Economic Policy, Applied Economics
Term
since 2021
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 464446725
This project will provide replicability rates for different literatures in economics. We will use a “Replication Market” as well as reproductions and additional robustness checks using the original data sets (henceforth R&R replications). The main objectives are, first, to define ‘replicability’ for R&R replications (the “What”) and, second, to identify reasons why replicability rates and market predictions thereof differ across study types that are frequently used in economics and social sciences (the “Why”).As for the “What” question, defining ‘replicability’ for R&R replications is an important and innovative first step. In economics and other social sciences, R&R replications of original data sets play a significant role in understanding replicability, because only a small share of the empirical literature is coming from standardized laboratory settings. Direct replications are rare and costly. Therefore, defining the success of R&R replications is pivotal to improve the replicability of empirical economics beyond the lab. We will develop this discussion further, by using our own experience with similar R&R replication studies, by reviewing previous R&R replications and by conducting an expert survey among experienced replicators in economics.As for the “Why” question, we proceed by R&R replicating 30 influential papers on similar topics in leading journals, 15 using Instrumental Variable (IV) methods, and 15 using Randomized Controlled Trials (RCTs). These are the two dominant techniques for causal inference in applied economics. We hypothesize both to have different a priori odds for replicability – which we will test in the Replication Market – because IVs are more prone to p-hacking and publication bias, whereas RCTs are often implemented in very specific samples. The major underlying problem for IVs is the virtual absence of pre-specification in secondary data-based economics studies. More specifically, in an R&R replication we expect IV-based papers to reveal lower replicability rates than RCTs, since p-hacking can be uncovered. The specific sample problem of RCTs, in contrast, cannot be detected in R&R replications. Hence to assess the generalizability across contexts, we will use a crowdsourcing approach such as the Bayesian Truth Serum in an expert survey. We will use these 30 R&R replications to incentivize the Replication Market that we will run on a larger number of 60 papers from the same literatures among experts in the field to come up with well-informed replicability predictions. Replication Markets are a specific form of prediction markets. We use this market and the expertise of researchers to deepen our understanding of the “Why” question in terms of p-hacking, publication bias, specification robustness and underpowered sample sizes.A particular feature of the project’s dissemination phase will be the interaction with practitioners and the broader public on the credibility of research on the science-policy interface.
DFG Programme
Priority Programmes
International Connection
Sweden, USA
Cooperation Partners
Professorin Anna Dreber Almenberg, Ph.D.; Professor Dr. Nathan Fiala; Professor Dr. Magnus Johannesson
