Multidimensional modelling of integrated academic-linguistic competences

• Applicants: Professorin Dr. Claudia Harsch; Professor Dr. Johannes Hartig
• Subject Area: General and Domain-Specific Teaching and Learning ; Applied Linguistics, Computational Linguistics
• Term: since 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 429641868

We apply for an extension of the project "Multidimensional modelling of integrated academic-linguistic competences", which currently running since 01/2020. Academic-linguistic competences in English are a decisive factor for a successful course of studies in international university programs. In order to assess these competences, authentic integrated task formats are a suitable option. Those tasks require text production based on previously read texts (reading into writing). The complexity of these tasks results in a higher complexity of the required competences, since several language skills such as reading and writing have to be used simultaneously. The aim of the project is to gain a better understanding of the academic-linguistic competences that underlie the successful performance in integrated tasks. The criteria for evaluating student text products and the judgment processes of the raters will also be investigated, as these are crucial for the quality of the competence measurements. In the current project, four integrated tasks and a scoring scheme were developed to rate the integrated writing products. Data were collected from N = 414 students and high school graduates who wrote 674 integrated writing products. Texts were evaluated by trained student raters. Qualitative data obtained from interviews with students and the evaluators provide important insights into writing and evaluation processes. Quantitative test data show, among other things, that the recorded performance is relatively independent of reading and writing skills measured with separate tests. The quality criteria show a somewhat different dimensional structure than assumed. Text quality assessed by humans can be predicted to substantial but varying degrees by text variables based on computational linguistic methods. A central problematic finding, however, is that the inter-rater agreements of the ratings for some criteria and raters are too low to achieve satisfactory reliability for single ratings of the texts. In the proposed extension, the existing texts will be re-evaluated by university teachers in order to repeat the quantitative analyses with an improved data basis. Simultaneously, this will allow a comparison of the judgments of raters with different levels of expertise. In addition, log data recorded during computer-based task processing will be analyzed in order to gain further insights into writing processes. Finally, analyses of the prediction of human judgments by computer-based text evaluations will be extended with methods of machine learning.

• DFG Programme: Research Grants