Gesture-Informed Language Models: Evaluating Multimodal Discourse Processing in LLMs and Humans

• Applicant: Dr.-Ing. Frances Yung
• Subject Area: Applied Linguistics, Computational Linguistics ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2026
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 579368214

Co-speech gestures provide important cues for understanding ambiguous discourse structure and pragmatic stances. We aim to investigate the fundamental question of whether, and how, LLMs can be enriched with gesture knowledge. We propose methods for representing gestures using both multimodal neural features and textual descriptions. We will develop neuro-symbolic methods, grounded in linguistic insights gesture studies,to improve discourse and pragmatic interpretation in multimodal interactions. In parallel, we will leverage LLM-assisted gesture annotation techniques to enable large-scale analysis of human gestural behaviour.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2556: Robust Assessment & Safe Applicability of Language Modeling: Foundations for a New Field of Language Science & Technology (LaSTing)