ITPC@VR: Process-Oriented Extension and Experimental Investigation of the Integrated Model of Text and Picture Comprehension to Explain Regulated Coherence Formation in Situated Learning with Virtual Reality

• Applicant: Dr. Christian Hartmann
• Subject Area: General and Domain-Specific Teaching and Learning
• Term: since 2026
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 579320219

The proposed project is situated in the field of multimedia learning and addresses a central challenge of immersive virtual realities: How can learners integrate verbal information with situational representations into a coherent mental model? While non-immersive multimedia learning environments support coherence formation through design features such as the spatial contiguity of text and images, virtual realities are visually complex, less structured, and potentially cognitively overwhelming. On the one hand, IVR environments open up new possibilities for vivid and situated learning; on the other hand, learners often struggle to consistently integrate verbal and visual information. Theories such as CTML or CAMIL capture these demands only to a limited extent. The ITPC model offers a process-oriented perspective but addresses coherence formation only implicitly and hardly considers the situatedness of virtual realities. The present project expands the ITPC model in a process-oriented way by conceptualizing coherence formation as regulated attentional control. Two strategies are distinguished: proactive coherence formation through anticipatory search for relevant visual information and reactive coherence formation through suppression of distracting stimuli. This differentiation makes it possible to explain heterogeneous findings and to further develop theories in a process-oriented manner. Empirically, the model extension will be tested in an experimental design. For this purpose, a 360° VR scenario of an operating room will be developed, placing medical students in a realistic learning situation in which they must integrate auditory with visual and digitally augmented information. Key variables include eye movements, cognitive load, knowledge acquisition, and transfer performance. By means of different task instructions (proactive vs. reactive), specific regulation strategies will be induced and their effects examined. This will reveal how learners regulate attention, when proactive or reactive processes are advantageous, and how they support the construction of coherent mental models. The project pursues three central goals: first, the theoretical advancement of the ITPC model; second, the empirical testing in a VR scenario; and third, the methodological establishment of temporal eye-tracking analyses as a standard for process-oriented investigations in multimedia learning. Theoretically, proactive and reactive coherence formation processes are described as central, innovative mechanisms of multimedia learning. Practically, the project provides recommendations for the design of coherence-supporting immersive learning environments. Thus, the project contributes to assessing the potential of virtual realities as an effective educational technology, sets new standards in process-oriented theory building, and enables the development of adaptive VR learning environments.

• DFG Programme: Research Grants
• Co-Investigator: Professorin Dr. Maria Bannert