Automated competency measurement and feedback generation for object-oriented programming

• Applicants: Professor Dr. Michael Goedicke; Professor Dr. Johannes Krugel
• Subject Area: Software Engineering and Programming Languages ; General and Domain-Specific Teaching and Learning
• Term: from 2019 to 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 412374068

Based on the course of the project so far, it has been possible to identify competence candidates through the didactic and empirical analysis of student programming code and in this way to link the automated analysis and evaluation of programming code as a solution to programming tasks with the evidence-based definition and measurement of competences for object-oriented programming. This lays the scientific foundation for creating corresponding assessment items and using them in teaching. Nevertheless, the successful use of these items requires the empirical validation of the competence candidates found so far and, in particular, the investigation of the effect of these items and the automatically generated, competence-oriented feedback based on them. In addition, the derivation of competence candidates can by no means be regarded as completed with the results so far, so that further competence candidates can and should be derived with the method successfully established during the project so far and associated items for competence measurement defined. The project thus pursues both didactic and educational technology goals. In the didactic part, the derivation of further competence candidates as well as the empirical validation of the found competence candidates are in the foreground, followed by the investigation of the mutual connection between automated feedback and the quality of the submitted solutions. The use of the items will generate extensive data that will allow conclusions to be drawn about knowledge gaps or misconceptions. The analysis of helpful feedback, in turn, will allow conclusions to be drawn about the nature and structure of the skills deficits that triggered the feedback, as well as about the quality and functioning of the items. Due to the project's duration of several years, a comparison of upcoming semesters and existing data sets will not only focus on short-term effects, but also on effects across semesters, in order to be able to observe wear-and-tear effects due to the use of identical tasks over several years. In the educational technology part of the objective, the focus is on the subject-specific further development of technical procedures, which concern both the support for the didactic analysis of large amounts of programme code and the automatic generation of specific, competence-oriented feedback. The aim is to automate the qualitative analysis of student programme code necessary for the didactic part to such an extent that mechanisms for recognising knowledge gaps, misconceptions and competence deficits can become an integral part of the e-assessment system JACK used and thus enable the further development of competence-oriented feedback beyond the end of the project term, especially in the everyday operation of a lecture.

• DFG Programme: Research Grants
• Co-Investigator: Dr. Michael Striewe