Selective reporting and the evolving research landscape in economics

• Applicants: Dr. Stephan Bruns; Professor Dr. Helmut Herwartz
• Subject Area: Statistics and Econometrics
• Term: since 2018
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 405039391

Replicability and, more generally, improving the reliability of empirical research is an increasingly important topic in economics. This project makes use of a comprehensive data set gathered by DORIS (Diagnosis Of Reporting errors In Scraped tables) with more than 600,000 hypothesis tests from the top 50 economics journals. First, we shed light on the prevalence and potential determinants of selective reporting in economics. Selective reporting refers to the selection of statistically significant findings in the research and publication process with the aim to increase the probability of getting published. Such a selection process may appear at the level of analyses (p-hacking) or at the level of entire studies (publication bias), and it should be emphasized that such a selection process is not necessarily a conscious manipulation of the research process but may occur as an unconscious “playing with the data”. We use discontinuity tests at the thresholds of statistical significance to elicit the presence of selective reporting. Our large sample size allows us to explore which covariates are associated with selective reporting (e.g., methods, sample sizes, authors' backgrounds, number of co-authors, field of research) and we also assess the (causal) effect of data and code availability policies on the prevalence of selective reporting by using a difference-in-difference design. The insights gained from this analysis can help to inform the research community, whether more or less measures to improve the reliability of empirical research are needed, and analyzing potential covariates may be of help in shaping such measures. Particularly, assessing the effectiveness of existing data and code policies will be of immediate use for journal editors to adjust, enforce or introduce such policies. Second, we empirically document how the research landscape in economics evolves by analyzing how research designs have changed in response to major trends. Specifically, we empirically document the “credibility revolution” by assessing whether methods applied in economics have changed towards (quasi-)experimental designs, but also whether economic research evolves in other aspects by documenting the number of control variables, the number of robustness checks and the use of clustered standard errors over time. Moreover, we also analyze the evolving landscape of economics research related to the “replication crisis” by documenting how many articles use pre-analysis plans, provide data and code, explicitly distinguish between exploratory and hypothesis-testing research, and how the reporting of null results change over time. Finally, we shed light on whether the “big data trend” is relevant in economics by documenting the evolution of sample sizes. Empirically documenting these trends will help the research community to take stock of the state of economics research and to further improve the reliability of empirical research in economics.

• DFG Programme: Research Grants