Test-taking engagement and test-taking behavior: Modeling the processes underlying item nonresponse and guessing
Final Report Abstract
Test-taking behavior – indicated among others by item-nonresponse or guessing – strongly impacts the results of cognitive assessments. The project focused on enhancing validity and comparability of cognitive assessment scores by modeling test-taking behavior and investigating the response process. We focused on modeling the speed-accuracy trade-off, itemnonresponse and guessing as well as on the distinction of engaged and disengaged testtaking behavior. We took different perspectives and propose models for the responses and response times in tests from different psychometric frameworks, including IRT-based models, accumulator models, and diffusion models. This allows for making different assumptions and depicting different aspects of the response process. First, we provide different models for evaluating the speed-accuracy trade-off in cognitive tests and their relation to effort: A growth curve model combining the 2-PL model with a log-normal factor model and a growth curve model based on the diffusion model. We found evidence for different change processes throughout a test (learning, adaptation) and for individual differences in the speedaccuracy trade-off. Second, we provide two models for investigating non-response being either influenced by previous test-taking behavior or by the amount of information accumulation: a model accounting for conditional dependence of item nonresponse and previous response process indicators and a process model based on an information accumulation process. These models improve over previous models as they relax strong assumptions about the interdependence between the responses, the response times and item non-response. We found evidence that previous responses as well as the ease of information accumulation in a specific item is related to item-nonresponse. Third, we developed models in order to disentangle engaged from disengaged response processes. Focusing on item omissions, we developed a model, in which we disentangle engaged from disengaged item nonresponse (omissions). The model also allows for incorporating different treatments of engaged and disengaged item omissions. Focussing on guessing, we developed models based on a race process representing information accumulation and motivational processes. Depending on the model version, disengaged responses are based on a random guess or an informed guess. Extensions of the models were developed that incorporate a decision between guessing and item non-response. We found that a minority of the test takers tends to disengaged responding throughout the test. These test takers guess randomly without using partial knowledge. Finally, we conducted an experimental study to validate the parameters of some of the models. The results support the claim that the speed-accuracy trade-off in cognitive tests is present and can be captured with growth curve models. The results for the race models indicate that test performance is related to capability and a persistence factor that determines the time the test takers spend on the item. The data of the study are published and openly available.
Publications
-
Disentangling Different Aspects of Change in Tests with the D-Diffusion Model. Multivariate Behavioral Research, 58(5), 1039-1055.
Ranger, Jochen; Wolgast, Anett; Much, Sören; Mutak, Augustin; Krause, Robert & Pohl, Steffi
-
Modeling speed-ability trade-off and test-taking persistence - Parameter validation for two psychometric models
Much, S., Mutak, A., Pohl, S. & Ranger, J.
-
A Model-Based Approach to the Disentanglement and Differential Treatment of Engaged and Disengaged Item Omissions. Multivariate Behavioral Research, 59(3), 599-619.
Ulitzsch, Esther; Zhang, Susu & Pohl, Steffi
-
Modeling the Intraindividual Relation of Ability and Speed within a Test. Journal of Educational Measurement, 61(3), 378-407.
Mutak, Augustin; Krause, Robert; Ulitzsch, Esther; Much, Sören; Ranger, Jochen & Pohl, Steffi
