Taking the Situation in Situational Judgment Tests Seriously – A Situational Construal Perspective

• Applicant: Professor Dr. Stefan Krumm
• Subject Area: Social Psychology, Industrial and Organisational Psychology ; Personality Psychology, Clinical and Medical Psychology, Methodology
• Term: from 2015 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 271360450

• Final Report Year: 2023

Final Report Abstract

Situational Judgment Test (SJT) items typically present realistic situations along with several response options. For a long time, SJTs were considered as simulative instruments in personnel selection (akin to assessment centers). However, Krumm et al. (2015) identified that a substantial number of items from several SJTs were solvable despite their core element, the situation descriptions, being eliminated—thereby fundamentally challenging the view of SJTs as simulations. In response to this, Lievens and Motowidlo (2016) proposed to view SJTs as methods that capture general domain knowledge (and not so much situational judgment). However, several authors opposed this notion. This is where the two herein reported two funding periods set in. One of the objectives was to determine whether the context-dependency of SJTs affected their validity. We revealed that the validity of SJTs was only marginally affected by the availability of situation descriptions. As another objective, we sought to identify the SJT item components that drive SJT context-dependency. To this end, we took a multi-study approach and first developed SJTs to specifically include trait-related information. We then manipulated the availability of this information in the situation descriptions. Indeed, we found that the availability of trait-related situational information affected the context-dependency of SJTs. As another main objective, we took a closer look at the (situational) processes involved when responding to SJT items. We revealed that test takers subjective construal of the situation predicted SJT response behavior, but that this prediction held up even if the situational construal was made only on the basis of SJT response options. We also identified—across two independent data collections and multiple SJTs—that the Ability to Identify Criteria (ATIC), which was found to be relevant in other simulations, was not a substantial driver of SJT performance. A similar finding was made for situational strength: SJT items’ situational strength did not moderate the relationship between personality and SJT performance. We then deconstructed SJT items into all their components and examined whether social desirability influenced responses to SJT item components. We revealed that results varied substantially across SJT items, but overall full SJT items were best predicted by deconstructed SJT items, in which only response options were shown. Social desirability had a significant effect on only a few items. In a final study, we examined whether several types of situation perceptions differed across SJT item components and predicted responses to these item components. We found that situation perceptions were overall poor predictors of performance across different SJT item versions. Overall, findings from two consecutive research grants suggest that response behavior in SJTs is complex, considerably different from other simulations, not very “situational”, and likely mostly influenced by response options and—if included—by trait-relevant information.

Publications

• Does taking away situations from situational judgment tests affect their validity? [Conference presentation] 10th International Test Commission Conference, Vancouver, Canada
  Schäpers, P.; Lievens, F.; König, C. & Krumm, S.
  
• How situational are video-based situational judgment tests? [Conference presentation] 32th Convention of the Society for Industrial and Organizational Psychology, Orlando, USA
  Schäpers, P.; Lievens, F. & Krumm, S.
  
• Removing situation descriptions from situational judgment test items: Does the impact differ for video‐based versus text‐based formats?. Journal of Occupational and Organizational Psychology, 93(2), 472-494.
  Schäpers, Philipp; Lievens, Filip; Freudenstein, Jan‐Philipp; Hüffmeier, Joachim; König, Cornelius J. & Krumm, Stefan
  
• Which kind of situational information is needed to make situational judgment tests situational? [Conference presentation] 19th Convention of the European Association of Work and Organizational Psychology, Turin, Italy
  Schäpers, P.; Lievens, F.; Freudenstein, J.-P.; Schulze, J.; König, C. J. & Krumm, S.
  
• Is it all in the eye of the beholder? The importance of situation construal for situational judgment test performance. Personnel Psychology, 73(4), 669-700.
  Freudenstein, Jan‐Philipp; Schäpers, Philipp; Roemer, Lena; Mussel, Patrick & Krumm, Stefan
  
• The role of Situations in Situational Judgment Tests: Effects on construct saturation, predictive validity, and applicant perceptions. Journal of Applied Psychology, 105(8), 800-818.
  Schäpers, Philipp; Mussel, Patrick; Lievens, Filip; König, Cornelius J.; Freudenstein, Jan-Philipp & Krumm, Stefan
  
• Auf der Suche nach der „Situationalität“ von Situational Judgment Tests [Searching the Situationality in Situational Judgment Tests]. 12 Biennial Conference of the German Psychological Society – Work and Organizational Psychology Section, Chemnitz, Germany
  Krumm, S.; Schäpers, P.; Freudenstein, J.-P.; Reznik, N. & Mussel, P.
  
• The Ability to Identify Criteria (ATIC) in Situational Judgment Tests [Conference presentation]. 16th Biennial Conference of the German Psychological Society - Personality Psychology and Psychological Diagnostics Section, Ulm, Germany
  Reznik, N.; Freudenstein, J.-P.; Heimann, A. L.; Ingold, P.; Kleinmann, M. & Krumm, S.
  
• Über die Relevanz der Bausteine von Situational Judgment Tests für die Testleistung und -validität [Conference presentation]. 52nd Conference of the German Psychological Society, Hildesheim, Germany
  Reznik, N.; Freudenstein, J.-P.; Schäpers, P.; Schulze, J.; Göritz, A. & Krumm, S.
  
• Does the ability to identify criteria predict situational judgment test responses? [Conference presentation]. 2023 SIOP Annual Conference, Boston, USA
  Reznik, N.; Krumm, S.; Freudenstein, J.-P.; Heimann, A. L.; Ingold, P.; Schäpers, P. & Kleinmann, M.
  
• The influence of situational strength on the relation of personality and situational judgment test performance. International Journal of Selection and Assessment, 32(1), 1-11.
  Freudenstein, Jan‐Philipp; Schäpers, Philipp; Reznik, Nomi; Stolte, Talea & Krumm, Stefan