Multimodal speech processing in static and dynamic cocktail-party situations

• Applicant: Professor Dr. Hartmut Meister
• Subject Area: Otolaryngology, Phoniatrics and Audiology
• Term: since 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 560945235

Verbal communication in everyday life involves several sensory modalities. In addition to auditory characteristics, visual features also play a crucial role, particularly for listeners with hearing impairment. This is especially relevant in situations with various interlocutors ("cocktail party"), where it is necessary to identify and separate the target talker from the background using multimodal information. However, the signals from different modalities may compete and affect each other. Furthermore, social gatherings such as cocktail party situations are often dynamic, as the target speakers typically change during conversation. The mechanisms at play in these complex situations and the cognitive load and effort involved in multimodal speech processing have not yet been fully clarified, particularly in the context of hearing impairment. The project aims to make a comprehensive contribution to closing this gap. To this end, multimodal scenes of varying complexity will be investigated, using both video recordings and simulations based on virtual reality. This allows modifying the scenes in a controlled manner and determining the consequences for speech processing at different levels. In addition to assessing speech intelligibility, this includes the recording of eye gaze as a measure of attentional focus and pupil size as a measure of cognitive load. This provides synchronized objective data, enabling a more precise view into the underlying mechanisms than was possible in most previous studies in this field. The project includes participants with varying degree of hearing impairment, listeners fitted with cochlear implants and a control group of subjects with good hearing ability. The overarching objective is to ensure a considerably better representation of real-life communication than what was previously possible through auditory-only investigations. This is achieved by taking multimodal dynamic cocktail-party situations into account. The aim is to uncover fundamental mechanisms in complex audiovisual scenes, describe differences between the various study groups, and establish new methodological approaches for stimulus presentation as well as comprehensive data recording. Based on this knowledge it is expected that improved clinical procedures will be promoted in the future, ensuring greater ecological validity and better individualisation of the diagnosis and rehabilitation of hearing impairment.

• DFG Programme: Research Grants
• International Connection: Belgium
• Cooperation Partner: Professorin Dr. Astrid van Wieringen