Multilevel Design Parameters for Sample Size Planning of Randomized Intervention Studies in Preschool, Elementary, and Secondary School

• Applicant: Professor Dr. Martin Brunner
• Subject Area: General and Domain-Specific Teaching and Learning ; Education Systems and Educational Institutions
• Term: since 2018
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 392108331

Education in preschool, elementary, and secondary school aims at fostering (cognitive) competencies and socio-emotional characteristics. There is an increasing need for educational research to provide evidence about which educational interventions and practices actually improve these outcomes. To provide this needed evidence, large-scale randomized experimental studies are indispensable because they allow for strong causal inferences on the effectiveness of interventions in ecologically valid settings. Such studies are often conducted as (a) individually-randomized trials (IRTs) or (b) cluster-randomized trials (CRTs), where individuals (e.g., students) or entire groups (e.g., entire schools), respectively, are randomly assigned to experimental conditions. To ensure that IRTs and CRTs are both sufficiently powered to detect intervention effects (if they exist) and (cost-) efficient, researchers must apply reliable design parameters to determine their sample size in an a priori power analysis. Given the multilevel organization of educational settings in Germany, design parameters involve information on the amount of variance in outcomes attributable to various levels (i.e., children/students, groups/classes, daycare centers/schools) as well as the amount of variance that can be explained at these levels by vital covariates, including socio-demographics and baseline measures. Our precursor project was the first project that provided such design parameters for students’ competencies in elementary and secondary school in Germany. Capitalizing on data from several German probability samples (e.g., the National Educational Panel Study [NEPS]), the present project substantively expands the knowledge base on the design parameters needed to plan the sample size of IRTs and CRTs in three ways: First, we analyze design parameters targeting the competencies and socio-emotional characteristics of children in daycare centers—the institutional preschool setting that most children attend in Germany. Second, we analyze design parameters targeting students’ socio-emotional characteristics in elementary and secondary school. Third, as it is impossible to provide design parameters with optimal fit to all potential outcomes and specific populations that may be targeted by IRTs or CRTs, we now develop a tutorial. This tutorial, involving all necessary statistical procedures, will enable educational researchers to take advantage of large-scale data sets (e.g., PISA) to estimate their own design parameters for continuous outcomes (e.g., scores on a standardized competence test) and binary outcomes (e.g., whether students earn a high school diploma or not) and to conduct power analyses for IRTs and CRTs. The tutorial will be accompanied by a workshop that empowers early career researchers to develop and analyze rigorous ICTs and CRTs to effectively and efficiently study the impact of education interventions.

• DFG Programme: Research Grants
• International Connection: USA
• Co-Investigators: Professorin Dr. Cordula Artelt; Professor Dr. Oliver Lüdtke
• Cooperation Partner: Professor Larry V. Hedges, Ph.D.