On the FLExibility and Stability of gesture-speecH coordination: Evidence from multimodal production, comprehension, and imitation (FLESH)

• Applicants: Aleksandra Cwiek; Dr. Susanne Fuchs; Dr. Wim Pouw, Ph.D.
• Subject Area: General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 502013782

Speech and gestures involve an orchestra of motions working in concert for communicative targets. Several previous studies have shown that speech and gestures are coordinated. However, it is less clear how stable or flexible coordination is and how coordination evolves. In the FLESH project, we assume that the coordination between speech and gesture is related to meaning and plays an essential role in the semiotic process. The originality of our approach is that we take motor properties into account. We suppose that stable gesture-vocal coordination needs to integrate the lighter mass and higher velocities of speech articulators compared to arm gestures. Moreover, in contrast to many previous theoretical approaches, we see coordination between speech and gesture as crucial for language evolution. To date, gesture studies and language evolution research did not detail motor control and gesture-speech coordination. Instead, researchers often focused on either vocalization or gesture, thus leaving out their coordination. In the FLESH project, we aim to change this by studying the coordination of vocalizations and gestures in relation to the expression of meaning. FLESH is organized into four work packages (WP). In WP1, we assess the natural motor properties of gesture and vocalization separately and in cooperation. We consider which coordination tendencies are relevant for understanding the meaning of gesture-vocal utterances. In WP 2, state-of-the-art technology and signal processing methodologies are used to investigate the coordination of the orchestra of multiple motions during imitation. We study the involved gesture-vocal movements with a high level of granularity to see whether particular types of motion are more likely to coordinate in meaningful versus non-meaningful imitations. In WP 3, we assess how gesture-vocal utterances evolve when passed on and imitated by a connected chain of imitators. This experiment will simulate language evolution in the lab. The focus will be on how coordination tendencies change or increase with the repetition of gesture-vocal utterances from one chain to the next. In WP 4, we will artificially change the temporal coordination between vocalization and gesture to see whether this affects semantic processing. Additionally, gesture-vocal utterances will be presented that do not match in terms of their meaning to see whether novel meanings emerge. The FLESH project will impact the theoretical understanding of gesture-speech relations, the relation between meaning and motor coordination, and its role in language evolution.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2392: Visual communication. Theoretical, empirical, and applied perspectives (ViCom)
• International Connection: Netherlands