Project Details
Projekt
Co-speech gestures and prosody as multimodal markers of information structure
Subject Area
General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
Term
since 2022
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 502013510
The project assesses how a crucial part of semantics and pragmatics, Information structure (IS) is encoded multimodally. While the interaction between IS and syntactic and prosodic patterns has been widely investigated, less is known about the multimodal/gestural marking of IS. We concentrate on the multimodal expression of focus, a cognitive category of information structure that identifies a constituent from a set of alternatives (Krifka, 2008). Focus highlights information, and we investigate whether increasing pragmatic prominence across focus types is expressed in increasing prosodic and gestural cues. The overarching aim of the project is to investigate multimodal focus marking in production, perception and cognition by exploring prosody and gesture in two typologically diverse languages, Catalan and German. The project’s first phase led to two major findings: that (a) the expression of focus types in both Catalan and German is multimodally encoded in terms of temporal alignment, prosodic and gestural prominence and gesture distribution, and that (b) gesture operates independently of prosody, indicating the presence of two distinct realizations for focus marking within the prosodic and gestural domains. Based on these findings, the primary objective of the project's second phase is to further explore the multimodal realization of focus within discourse through three distinct yet complementary perspectives: production, perception, and cognition. The first three strands of the project will address these domains as follows. Strand 1 will enhance the findings from production data by thoroughly evaluating the relationship between acoustic and visual cues in the marking of focus types in the temporal and spatial domains. Strand 2 will assess the role of gesture and prosody from a perceptual perspective, specifically studying the contribution of gestural and prosodic cues to the comprehension of focus types. Strand 3 will evaluate the cognitive utilization of gestures in discourse structuring, informed by the Gesture-for-Conceptualization Hypothesis (Kita et al., 2017). This investigation will determine whether gesture prominence significantly influences self-oriented cognitive functions during discourse production, particularly in activating, manipulating, packaging, and exploring discourse information. Finally, Strand 4 addresses a central methodological objective of the project, which pertains to the advancement of the concept of gestural prominence. This will be achieved through the evaluation of kinematic features of gestures, which will contribute to the refinement of strategies for annotating gesture prominence. All in all, we expect to gain crucial insights that will contribute to the development of more nuanced models of multimodal language processing and production as well as a comprehensive view on the pragmatics-gesture-prosody interface.
DFG Programme
Priority Programmes
Subproject of
SPP 2392:
Visual communication. Theoretical, empirical, and applied perspectives (ViCom)
International Connection
Spain
