Do automatic validation and reactivation processes also take place when learning with text and pictures?
Final Report Abstract
Text-picture combinations play a central role in knowledge acquisition. The empirically observed learning-enhancing effect of text-picture combinations compared to text presentation alone is explained by the fact that learners select relevant information, organize it into both a verbal and a pictorial mental model, and then integrate these models. Based on findings from text comprehension research, it can be assumed, however, that beyond selection, organization, and integration, additional cognitive processes might play a central role when learning with text and pictures. These include the passive reactivation and validation processes postulated in the RI-Val Model by Cook and O'Brien (2014). The reactivation process reactivates information that is no longer active in working memory but overlaps with currently held information. This reactivated information is then integrated with the current information. The validation process checks whether this integration is consistent against the background of world knowledge or the discourse context. Both processes are passive in nature and occur automatically. In the DFG-funded project, it was investigated whether automatic cross-modal reactivation and validation processes occur during the processing of text-picture combinations— specifically, whether picture content is reactivated when reading a text and vice versa, and to what extent the compatibility of text and picture content is validated. In a series of experiments using the epistemic Stroop paradigm, we were able to demonstrate in four experiments that cross-modal validation processes, as well as cross-modal reactivation and validation processes occur when using both single-sentence material and multi-sentence material combined with pictures. The advantage of the Stroop paradigm lies in the isolated investigation of the processes, while its disadvantage lies in the artificial nature of the material. In another series of experiments, we, therefore, used the contradiction paradigm. In these experiments longer illustrated texts were presented in either a consistent (i.e., without contradiction) or inconsistent version (i.e., a target sentence contradicted the previous discourse context). Using reading times and reaction times in test questions in online studies, as well as eye-tracking in the lab, we investigated whether cross-modal reactivation and validation processes also play a role in more complex illustrated texts. The reading times and reaction times suggested that text and/or picture content was reactivated during text processing, but this could not be demonstrated in the eye-tracking experiments – at least at the current stage. In a last experiment, we investigated whether pictures support the reactivation and integration of outdated and actual information when processing refutation texts. Here, it seems that pictures have no additional advantage. In sum, this research project fills a previously existing research gap by demonstrating crossmodal, passive reactivation, and validation processes in the processing of multimedia. Exploring the processes underlying learning is central to deriving practical recommendations for the design of multimedia.
Link to the final report
https://doi.org/10.23668/psycharchives.21160
Publications
-
Reaktivierungs- und Validierungsprozesse beim Lernen mit Text und Bild. 18. Tagung der Fachgruppe Pädagogische Psychologie (PaePsy) 2021. Pädagogische Hochschule Heidelberg. [Runder Tisch]
Frick, P. & Schüler, A.
-
Does the RI-Val model also apply to illustrated texts? 32nd Annual Meeting of the Society for Text & Discourse (ST&D). Georgia State University. Atlanta, GA, USA. [Talk]
Frick, P. & Schüler, A.
-
Not just texts – reactivation, integration, and validation processes also apply to illustrated texts. European Association for Research on Learning and Instruction (EARLI). Special Interest Group (SIG) 2. Kiel. [Talk]
Frick, P. & Schüler, A.
-
Text-picture combinations are integrated and validated automatically. 64. Tagung experimentell arbeitender Psycholog:innen (TeaP) - Conference of Experimental Psychologists. University of Cologne (online conference). [Talk]
Frick, P. & Schüler, A.
-
Dataset and code for: Extending the theoretical foundations of multimedia learning: Activation, integration, and validation occur when processing illustrated texts
Frick, P. & Schüler, A.
-
Extending the theoretical foundations of multimedia learning: Activation, integration, and validation occur when processing illustrated texts. Learning and Instruction, 87, 101800.
Frick, Pauline & Schüler, Anne
-
Kognitive Prozesse bei der Verarbeitung von Text- Bild-Kombinationen. Eingeladener Vortrag auf der Tagung Schrift / Bild – Lesen. Interdisziplinäre Perspektiven. Universität zu Köln
Schüler, A.
-
Reading research meets multimedia: textpicture combinations are validated automatically. 20th Biennial Conference of the European Association for Research on Learning and Instruction (EARLI). Thessaloniki, Greece. [Talk]
Schüler, A. & Frick, P.
-
Specifying the cognitive processes underlying the processing of multimedia materials. Invited talk at the Universitá degli Studi di Padova. Padua, Italy
Schüler, A.
-
The cognitive processes when reading and learning with illustrated texts. Winter School Eye Tracking - Experimental Design, Implementation, and Analysis. ETH Zürich, Monte Verita, Switzerland. [Poster]
Frick, P. & Schüler, A.
-
Cross-modal activation, integration, and validation processes when reading illustrated texts: An eye-tracking study. Visual Language (VisLang) Conference 2024. Tilburg University, Netherlands. [Talk]
Frick, P., Hoch, E. & Schüler, A.
-
Cross-modal validation processes occur during processing multimedia materials. European Association for Research on Learning and Instruction (EARLI). Special Interest Group (SIG) 2. Valencia, Spain. [Talk]
Schüler, A. & Frick, P.
-
Dataset and code for: Do passive cross-modal validation processes occur when processing multimedia materials?
Schüler, A. & Frick, P.
-
Do passive cross-modal validation processes occur when processing multimedia materials?. Learning and Instruction, 93, 101956.
Schüler, Anne & Frick, Pauline
-
Synthesizing text comprehension and multimedia learning research: text-picture combinations are activated, integrated and validated automatically. 66. Tagung experimentell arbeitender Psycholog:innen (TeaP) - Conference of Experimental Psychologists. University of Regensburg. [Talk]
Frick, P. & Schüler, A.
