Assessing Students’ Texts: German (ASSET-G) weiter als Training Assessment Competencies in German (TrACE-G )

• Applicants: Dr. Thorben Jansen; Professor Dr. Jörg Kilian, from 6/2021 until 7/2025; Professor Dr. Jens Möller
• Subject Area: General and Domain-Specific Teaching and Learning ; Developmental and Educational Psychology
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 456083968

TrACE-G (Training Assessment Competencies in German) investigates determinants of teachers’ diagnostic judgments and their influence on the assessment of students' texts. TrACE-G is a direct follow-up to ASSET-G (Assessing Students' Texts – German) and connected with two D-A-CH projects: ASSET (Assessing Student's English Texts) and TrACE (Training Assessment Competences in English as a second language). While ASSET and ASSET-G analyzed teacher evaluations of texts in English and German, mainly in experimental studies, TrACE adds a comprehensive analysis of authentic student texts and interventions to improve teacher judgment accuracy in English. TrACE-G continues this work for German texts and investigates whether increasing the salience of judgment-relevant text features defined in a scoring rubric using automated highlighting tools based on Large Language Models (LLM) can focus the assessment process on the highlighted text feature, thereby increasing the accuracy of judgments and reducing bias. The main research questions are: 1. Which teacher, text, and judgment characteristics relate to teacher judgment accuracy (TJA) and judgment biases in the context of first-language report writing in German? 2. How accurately can judgment-relevant content aspects and features within reports be automatically highlighted compared to expert annotations? 3. Does increasing the salience of judgment-relevant text features by automatic highlighting tools increase these features’ importance in the judgment process and thus increase the accuracy of judgment and reduce judgment bias? Our work begins with a comprehensive investigation of which text and student characteristics contribute to the diagnostic process, judgment accuracy, and biases. Working with a large corpus of authentic student texts from the IQB makes it possible to investigate determinants of judgment accuracy and bias within a single study. In this Study 1, the influence of teacher, text, and judgment characteristics on teacher judgment accuracy will be determined. We will then develop the automatic highlighting of judgment-relevant text features such as spelling errors, content aspects, and aspects of the text type report as a manipulation of salience using LLM (Study 2) and compare the annotations with those of experts to ensure the quality of the automated annotations (Study 3). The usage of LLMs enable to transfer the automated annotation to other text types than reports. In Study 4, we will investigate whether teachers who receive texts with automatically annotated text features take them into account more in their assessment process and thus become more accurate in their assessment and reduce bias.

• DFG Programme: Research Grants
• Co-Investigators: Professorin Dr. Andrea Horbach; Privatdozent Dr. Simon Tiffin-Richards