Developing a cross-methodlogical processing account of situated language comprehension

• Applicant: Professorin Dr. Pia Knoeferle
• Subject Area: General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
• Term: from 2006 to 2008
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 29734138

• Final Report Year: 2009

Final Report Abstract

The project "A cross;methodological approach to developing a processing account of situated language comprehension" investigated the time course and neural correlates of visual context effects on online sentence comprehension using eye tracking and ERPs. Previous research has shown rapid and incremental visual context influences on language comprehension. The project research was motivated by the potential relevance that more detailed insights into the time course and nature of such visual context effects bear for developing theories of sentence comprehension. The present research investigated (1) the time course of visual context effects when-picture and language are often incongruous. In more detail, this part of the project examined variation in the time course and nature of visual context influences on online language comprehension as a function of (1a) verbal working memory; (1b) different kinds of information in visual context; (1c) age, (1d) task, and (1e) event plausibility as well as event presentation (static vs dynamic). In addition, we examined how motor knowledge affects the time course of visual attention and online comprehension. A computational implementation made first steps towards accounting for both behavioural and neural correlates of situated comprehension. The key results were that the time course of visual context effects varies as a function of (a) verbal working memory (delayed for participants with low relative to high verbal working memory), age (delayed for older relative to younger adults), the kinds of information in visual context (visual context effects related to who-does-what-to-whom preceded verb-action congruence effects), and event plausibility (earlier for plausible than non-stereotypical events). The extended CIANet model accounted not only for the time course (implementing visual anticipation of relevant role fillers) but in addition also for the neural correlates of sentence comprehension in relevant non-linguistic visual context.

Publications

• (2007). Incremental interpretation of motor knowledge affects the time course of visual attention during situated comprehension. AMLAP, Turku, Finland
  Knoeferle, P. & Steinberg, J.
  
• (2008) ERP correlates of verb-action and sentence-scene role relations incongruence in a sentence-picture verification task. In: Psychophysiology 45, 90-90
  Knoeferle, P., Urbach, T. & Kutas, M.
  
• (2008). ERP correlates of verb-action and sentence-scene role relations in a sentence-picture verification task. In: Proceedings of the Annual Meeting of the Cognitive Neuroscience Society (p. 251). San Francisco, CA
  Knoeferle, P., Urbach, T. & Kutas, M.
  
• (2008). Incremental syntactic disambiguation using depicted events: plausibility, co-presence and dynamic presentation. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp.2398-2403). Austin, TX: Cognitive Science Society
  Ellsiepen, E., Knoeferle, P., & Crocker, M.W.
  
• (2008). The influence of plausible scene events during incremental syntactic disambiguation. In Proceedings of the 21th CUNY Conference on Human Sentence Processing, Chapel Hill, USA
  Ellsiepen, E., Crocker, M., & Knoeferle. P .
  
• (2009). A neurobehavioral model of situated language comprehension. CUNY 2009
  Crocker, M.W., Mayberry, M., & Knoeferle, P.
  
• (2009). What can sentence-picture verification reveal about visual context infiuences on online language comprehension? CUNY 2009
  Knoeferle, P., Urbach, T., & Kutas, M.