Project Details
Projekt
Computational Semantics of Graded Speech–Gesture Integration Using VR Technology (CoSGrIn-Vr)
Applicants
Dr. Andy Lücking; Professor Dr. Alexander Mehler
Subject Area
Applied Linguistics, Computational Linguistics
Methods in Artificial Intelligence and Machine Learning
Methods in Artificial Intelligence and Machine Learning
Term
since 2022
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 502018965
The CoSGrIn-VR project addresses the computational semantic mechanisms underlying speech-gesture integration, a research gap in formal semantics and multimodal communication studies. Little is known in computational semantics about how gestures acquire meaning and how gesture meaning combines with speech meaning. The project challenges holistic and gestalt-based models by developing a computational semantics of speech-gesture integration that is (1) formally modelled, (2) computationally implementable, (3) experimentally testable in virtual reality (VR), and (4) cognitively interpretable. CoSGrIn-VR focuses on two key research areas: (i) developing computational semantic models for recognizing acting gestures and (ii) extending the GeMDiS-Model from the first phase to handle graded exemplification, addressing cases where gestures only partially or indirectly affiliate with speech. The first research area examines how perceptual action classifiers can account for acting gestures, which simulate actions. The second research area investigates the informational uncertainty in gesture-speech integration by studying quantified noun phrases and references to atypical objects. Methodologically, the project employs Interaction Verification Games within VR environments to test its models. These immersive experiments capture multimodal data, including body movements, gestures, spatial behavior, and gaze. Additionally, CoSGrIn-VR aims to develop an AI-based VR lab to enhance experimental control using avatars and automate the annotation of multimodal communication. A key component of this approach is the Multi-perspective Annotation Model (MAM), which enables the systematic and largely automatic annotation of multimodal experimental data. This integration of linguistic theory, computational modeling, and VR-based empirical testing positions CoSGrIn-VR as a novel approach to understanding the graded nature of speech-gesture integration and its implications for visual communication.
DFG Programme
Priority Programmes
