Explaining Inconsistent Effects of Teaching Quality on Educational Outcomes

• Applicants: Dr. Carmen Köhler; Dr. Thomas Lösch
• Subject Area: General and Domain-Specific Teaching and Learning
• Term: since 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 518236946

The focus of educational effectiveness research is to identify aspects of teaching that foster student learning outcomes. The ultimate goal is to inform policy makers and practitioners in the field about teaching practices and educational initiatives that improve student learning. Various meta-analyses within the research field exist. They show large deviations in effect sizes across studies that investigated the same substantial research question, which implies replication failure. Inconsistent and, in part, even contradictory results hinder straightforward recommendations and raise doubts about the credibility of the research results. Reasons for deviations could be that the studies vary with respect to several aspects: (a) the setting, (b) the design, (c) the operationalization of the constructs, and (d) the methodology used in the statistical analyses. Meta-analyses and reviews showed that some of those aspects influenced the estimated effect of teaching quality on student outcome. Since the meta-analyses and reviews gathered information from already existing studies, however, they were unable to keep certain aspects such as the statistical model constant, and were thus unable to disentangle the different aspects that are responsible for heterogeneous effects. Conducting a systematic re-analysis of already existing data, our two main aims are (1) to identify and precisely define possible factors that might explain variance in effects of teaching quality, and (2) to provide estimates of the relative importance of those factors for heterogeneity of results. In pursuit of the first research objective, we will categorize data sets with respect to differences in (a) the design and setting and (b) the operationalization of constructs. We will further define methodological factors that might influence the estimated effect. To achieve the second research objective, we will harmonize the data sets, run the secondary analyses using different methodological approaches, extract the estimated effects from the individual analyses and transform them onto the same scale (in the form of an effect size), and estimate which aspects explain which proportion of variance in the effects. In the end, these findings aid in the planning of future generalizability analyses and conceptual replications that result in meaningful recommendations for practitioners.

• DFG Programme: Research Grants
• International Connection: Switzerland
• Co-Investigator: Professor Dr. Johannes Hartig
• Cooperation Partner: Professorin Dr. Anna-Katharina Praetorius