Synergia: Multisensory Integration in High-Intensity Environments — Bridging AI Analysis and Human Perception

• Applicants: Professorin Merle T. Fairhurst; Privatdozent Dr.-Ing. M. Ali Nasseri; Professor Dr. Nassir Navab
• Subject Area: Biomedical Systems Technology
• Term: since 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 560596849

The impact of human multisensory processing in highly complex tasks and the influence of environmental factors remain largely unexplored. Fundamental questions persist, as most multisensory research has been confined to simple experimental setups. However, recent technological advancements have made extensive multimodal digital data accessible, which can now be processed by intelligent, context-aware systems. By leveraging artificial intelligence and machine learning, these systems can semantically fuse and analyze data in real-time, enabling them to perceive the environment and localize and control tools with increasing precision. This provides deeper insights into complex real-world environments. The outputs of AI systems have been conventionally presented to users through unimodal, predominantly visual, formats via numerous displays and devices. This reliance on visual channels can lead to oversaturation and cognitive overload, diminishing the usability of AI technologies. In contrast, multisensory integration offers a promising solution by distributing perceptual demands across multiple senses, thereby enhancing usability and reducing cognitive strain. To address these limitations, this proposal aims to conduct a systematic investigation into the contributing components of multisensory environments. Specifically, we focus on understanding human multisensory processing within the high-intensity context of surgical procedures—an ideal setting for studying complex tasks under demanding conditions. Our goal is to design multisensory surgical navigation systems with a particular focus on integrating auditory feedback models to complement and enhance existing visual-based computer-assisted systems. With such a multisensory system in place, we will investigate human performance and perception in high-intensity surgical tasks, considering factors such as information complexity, training, and expertise. These investigations will take place in both controlled lab settings and more ecologically valid environments. This interdisciplinary effort will bring together experts in human multisensory perception, AI engineering, and medical professionals to explore and develop innovative methodologies for multisensory design in the complex and high-stakes environment of surgical procedures.

• DFG Programme: Research Grants