TRACE - Training Assessment Competences in English as a Second Language

• Applicant: Professor Dr. Jens Möller
• Subject Area: Educational Research on Socialization, Welfare and Professionalism ; General and Domain-Specific Teaching and Learning ; Developmental and Educational Psychology
• Term: since 2016
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 315271436

TrACE (Training Assessment Competencies in English as a Second Language) is a follow-up to two D-A-CH projects: ASSET (Assessing Student’s English Texts) and MEWS (Measuring English Writing at Secondary Level). While ASSET was concerned with teachers’ diagnostic competences when judging English student texts, MEWS generated a large corpus of authentic learner texts with holistic ratings by trained humans and assessment software. TrACE combines these projects to achieve following objectives: 1) Generating detailed ratings of text qualities for a large corpus of authentic student texts, achieving highest possible objectivity by combining human ratings and machine scoring (Studies 1 & 2); 2) Investigating what factors at teacher, student, text, and judgment level determine the ability of preservice and in-service English teachers to assess complex English learner texts accurately by comparing their assessment to human- and machine-made benchmark scores (Study 3);3) Building a comprehensive, free online training tool and evaluating its effectiveness with a large sample of pre-service English teachers (Study 4, 5 & 6).Empirical studies have shown scoring consistency to be relatively high when raters use analytic instead of holistic assessment practices (i.e., apply detailed criteria to student texts) (Dempsey et al., 2009). Therefore, in Study 1, human raters will assess a large portion of texts from the MEWS corpus using analytic criteria developed in ASSET, relating to aspects of textual quality typically assessed by teachers in schools. These ratings serve as the basis for Study 2, in which automatic scoring models for the analytic human scores are developed using natural language processing techniques. Working with a large corpus of authentic texts will allow us to investigate key determinants of judgment accuracy as continuous variables within one single study. Study 3 will investigate the influence of teacher, student, text, and judgments characteristics on teacher judgement accuracy (Südkamp et al., 2012). The same corpus is then used to investigate the effectiveness of specific training measures: Study 4 will examine whether teachers who receive feedback on their analytical text judgments (comparison with benchmarks) become more accurate in their assessment. Studies 5 & 6 will provide teachers with automatically generated visualized feedback on key features of text quality to determine whether this helps them to assess texts more accurately (vocabulary in Study 5 and argumentative structure in Study 6). Based on the results of these studies, in TrACE we will develop a comprehensive, free on-line training tool for pre-service and in-service teachers to develop their diagnostic competence in judging student texts while using detailed analytic criteria.

• DFG Programme: Research Grants
• International Connection: Switzerland
• Cooperation Partner: Professor Dr. Stefan Keller