Software-based Evaluation of Open-ended items

• Applicant: Professor Dr. Hendrik Härtig
• Subject Area: General and Domain-Specific Teaching and Learning
• Term: from 2013 to 2016
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 250560060

Currently the professional knowledge of (prospective) teachers is in the focus of the research community. Within the commonly used paper-pencil instruments one may include closed and open-ended items. In open-ended items the students are not influenced by distractors, general solution strategies will also not help. Furthermore, the underlying solution processes can be better assessed by using open-ended items. Students use adequate and naïve ideas in parallel in open-ended items, which one could not find using multiple-choice questions. However the more students shall be assessed, the less open-ended items are aligned with the test economics. It is often difficult to achieve a reasonable inter-rater reliability. This led to the dominance of the closed items, for example in typical concept inventories.Due to technical developments, computer-based evaluation (CBE) is carried out assessing students' skills. Additionally open-ended items can be evaluated directly by means of a software-based evaluation (SBE). Due to positive experiences in the field of linguistic abilities, first experiments regarding scientific understanding exist. It is shown that with a sufficient sample size, the inter-rater reliability between software and human ratings equal to higher than between the experts. The discussed above disadvantages of open-ended items, might be compensated by using the SBE.Based on a cooperation opportunity with the KiL project, in the project applied for here it will be evaluated the extent to which SBE is possible and efficient. The KiL project deals with the acquisition of professional knowledge of teacher training students in math and science. Pedagogical knowledge, content knowledge and pedagogical content knowledge are assessed. What is needed is an appropriate number of open-ended items. The items used in the context of the KiL project will already be piloted and optimized concerning inter-rater reliability.

• DFG Programme: Research Grants
• Participating Person: Professor Dr. Thilo Kleickmann