Project Details
Projekt
Applying Principles from the Validity-Based Framework of Replication in Metastudies to Test Causal Relations
Applicant
Dr. Marc Jekel
Subject Area
General, Cognitive and Mathematical Psychology
Social Psychology, Industrial and Organisational Psychology
Social Psychology, Industrial and Organisational Psychology
Term
since 2021
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 464547651
Metastudies have been proposed as a method to assess the replicability and generalizability of experimental findings by identifying moderators in causal relationships. This method involves conducting a series of small-scale studies, each modifying the properties of tasks and including more diverse participant samples. Known as radical randomization, this approach moves away from the fixed properties of manipulations and measures and the limited variability in participant characteristics typical in conventional experimental studies. Its application has successfully provided a more nuanced understanding of the phenomena under study, thereby aiding in theory refinement and development. In our proposal, we aim to expand the scope of metastudies by incorporating aspects from the validity-based framework of replication that have been underrepresented in the current approach. We focus on two key aspects in our extension of metastudies: the effectiveness of manipulations in actually manipulating the intended construct, and the functional relationships between the independent and the dependent variable. Our demonstration of the extension will concentrate on two frequently manipulated constructs: intuitive cognitive processing and affect, and we will test their causal relationship with cooperation behavior. Replication studies and meta-analyses have shown mixed results for the relationship between intuition and cooperation, which might be partially attributed to validity issues. Specifically, to assess the construct validity of manipulations, we will examine the effectiveness of various manipulations of intuition and affect in dedicated pre-studies and incorporate these manipulations into our metastudy. To enhance statistical conclusion validity, we will utilize a recently proposed machine-learning approach. This approach can, in principle, capture any relationship between manipulations and measures, offering an advantage over traditional statistical modeling frameworks with default (often linear) functions. Utilizing the results from our metastudy, we plan to use computer simulations to forecast replication rates and replication effect sizes for our exemplary causal relationship. To further validate our findings, we have planned a replication study to test predictions about effect sizes derived from these simulations. Our proposal aims to understand why effect sizes vary in replication studies and how to devise methods for predicting these sizes based on the properties of manipulations and measures, participant characteristics, and their functional relationships with the dependent variable.
DFG Programme
Priority Programmes
Co-Investigator
Dr. Dorothee Mischkowski
