Project Details
Projekt
The role of measurement in the replicability of empirical findings
Applicant
Professorin Dr. Eunike Wetzel
Subject Area
Personality Psychology, Clinical and Medical Psychology, Methodology
Term
since 2021
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 464394046
The goal of this project is to investigate the role of measurement in the replicability of empirical findings. We will focus on two widespread, potentially problematic measurement practices: 1) the use of ad hoc scales and 2) the use of modified scales. Ad hoc scales are scales that authors constructed for a particular study and that have not (or only very superficially) been validated. The term “modified scales” refers to deviations from preexisting, validated scales for example in terms of the number of items, the items’ wording, or the response format. We will investigate how these practices influence replication rates and the heterogeneity of effect sizes using three complementary methodological approaches: by reanalyzing existing empirical data, running an experiment, and conducting a simulation study. Study 1a will reanalyze data from replication projects that applied item-based measures such as the Many Labs projects. We will code the measure type (validated vs. ad hoc, different types of modifications) used in the original study and the replication studies and investigate whether replicability differs between measure types. Study 1b will systematically manipulate the measurement properties of item-based measures in the GESIS Panel data set and investigate how this affects the replicability of effects reported with these data. Study 2 will conduct a multisample replication project on multiple original effects from different fields that is combined with an experiment on the influence of measurement on replicability and effect size heterogeneity. Study 3 will consist of a simulation study in which the “original” scale will be compared to ad hoc scales and different types of modifications of the original scale in terms of the recovery of the true effect sizes. In Study 2, we will focus on modifications whose effects are best studied empirically, such as rewording items and changing the response format, whereas in Study 3, we will focus on modifications that can be simulated well, namely reducing the number of items. We will use multiple indicators to determine replication success, the most important one being a Bayes factor that compares the null hypothesis with the alternative hypothesis for each effect. Replication rates will then be compared across measure types. Furthermore, we will quantify effect size heterogeneity using a meta-analytic approach and include measure type as a moderator. Our application context is multidisciplinary and includes several areas within psychology (social psychology, health psychology) as well as the political sciences and economics, allowing us to make comparisons across fields. In sum, this project will contribute to the META-REP priority program and the metascientific literature by investigating measurement as a factor that can explain the replicability of empirical findings.
DFG Programme
Priority Programmes
International Connection
USA
Cooperation Partner
Professor Dr. Brent W. Roberts, Ph.D.
