Development of mixed distribution item response models for the analysis of cross-classified multirater data and their application in teaching evaluation research (MixIRT-CCM)

• Applicant: Professor Tobias Koch, Ph.D.
• Subject Area: Personality Psychology, Clinical and Medical Psychology, Methodology
• Term: from 2018 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 405463675

• Final Report Year: 2024

Final Report Abstract

The project’s main objective was to develop models for cross-classified multirater data, where multiple raters rate multiple targets on items with ordered response categories. Moreover, it is assumed that the rater population is heterogeneous and consists of latent subpopulations (mixture distribution assumption). The resulting models are mixture cross-classified item response theory models. Two version were developed: a graded response model and a generalized partial credit model. All models were defined on the principles of stochastic measurement theory and fit via Bayesian estimation by means of the open-source software Stan. For model assessment and comparison, leave-one-out cross-validation as well as stacking was used. The model code is publicly accessible and contains an integration algorithm to calculate the marginalized likelihood that can be easily adapted to similar models. The analysis of teaching evaluation data reveals two latent rater populations that show extreme vs. moderate response styles. These results suggest that the heterogeneity of the student population in teaching evaluation research should be considered. The results of a simulation study show that the parameters can be satisfactorily recovered in the developed models.

Publications

• Introducing Bayesian mixture crossclassified IRT models [Konferenzbeitrag]. 15. Konferenz der Fachgruppe für Methoden & Evaluation, Mannheim.
  Bee, R. M. & Koch, T.
  
• Identification of composed CFA models – A tutorial on how to identify bifactor and related models which include latent criterion variables [Konferenzbeitrag]. 52. Kongress der Deutschen Gesellschaft für Psychologie, Hildesheim.
  Bee, R. M., Koch, T. & Eid, M.
  
• A General Theorem and Proof for the Identification of Composed CFA Models. Psychometrika, 88(4), 1334-1353.
  Bee, R. Maximilian; Koch, Tobias & Eid, Michael
  
• Managing the Intricacies of Teaching Evaluation Data with Mixture Cross-Classified Item Response Theory Models.
  Bee, R. Maximilian & Koch, Tobias