Brain Dynamics in Cyber-Physical Systems as Measure of User Presence

• Applicant: Professor Dr. Klaus Gramann
• Subject Area: Human Factors, Ergonomics, Human-Machine Systems
• Term: from 2021 to 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 462163815

• Final Report Year: 2025

Final Report Abstract

This project aimed to develop a multi-modal metric for assessing presence in extended reality (XR) environments. Presence—the feeling of truly "being there" in a virtual world—was explored by analyzing how the brain (via EEG) and body (through motion capture) respond to virtual interactions, particularly when sensory feedback is either consistent or inconsistent. In a series of studies, we utilized EEG, peripheral physiological markers, motion capture, and performance to examine how different feedback modalities shape presence. One major finding was that sensory mismatches influenced brain activity in specific frequency bands, altering sensorimotor processing. Additionally, brain signals related to error detection in the anterior cingulate cortex (ACC) could classify unrealistic VR interactions with 77% accuracy, highlighting the ACC’s role in detecting sensory inconsistencies and adjusting behavior. Contrary to our expectations, we did not observe significant effects of sensory mismatches on peripheral physiological measures such as ECG and skin conductance, likely due to the short time windows of analysis and participant adaptation. Another deviation was the inability to validate the widely used presence questionnaires (IPQ and SUS) due to participants focusing on the novel experience of the haptic glove. Instead, we used subscales from the Multimodal Presence Scale, which provided more relevant insights into the realism of the VR experience. Additionally, force-feedback synchronization issues led us to shift focus from sensory noise to the Sense of Agency (SoA), exploring it through a study using muscle stimulation to control participants' movements. A key achievement was the development of a neuroadaptive XR system. By integrating neural decoders with reinforcement learning (RL), the system dynamically adjusted haptic feedback based on brain activity, achieving high accuracy in classifying user experience. This validated an unobtrusive method for real-time presence adaptation. To ensure research transparency, data recorded in this project were structured following the Brain Imaging Data Structure (BIDS) standard, and an EEG-motion dataset was made openly available. Additionally, one workshop on BIDS curation and one on presence measurement in XR brought together experts to discuss advancements and challenges to discuss data curation and presence in XR, respectively. This project contributes to neuroadaptive XR research by providing insights into presence perception, real-time adaptation, and the limitations of physiological measures, paving the way for future immersive VR advancements.

Publications

• Neural sources of prediction errors detect unrealistic VR interactions. Journal of Neural Engineering, 19(3), 036002.
  Gehrke, Lukas; Lopes, Pedro; Klug, Marius; Akman, Sezen & Gramann, Klaus
  
• The BeMoBIL Pipeline for automated analyses of multimodal mobile brain and body imaging data. openRxiv.
  Klug, M.; Jeung, S.; Wunderlich, A.; Gehrke, L.; Protzak, J.; Djebbara, Z.; Argubi-Wollesen, A.; Wollesen, B. & Gramann, K.
  
• Toward Human Augmentation Using Neural Fingerprints of Affordances. Affordances in Everyday Life, 173-180. Springer International Publishing.
  Gehrke, Lukas; Lopes, Pedro & Gramann, Klaus
  
• Towards an Implicit Metric of Sensory-Motor Accuracy: Brain Responses to Auditory Prediction Errors in Pianists. Creativity and Cognition, 129-138. ACM.
  Pangratz, Elisabeth; Chiossi, Francesco; Villa, Steeven; Gramann, Klaus & Gehrke, Lukas
  
• Decoding Realism of Virtual Objects: Exploring Behavioral and Ocular Reactions to Inaccurate Interaction Feedback. ACM Transactions on Computer-Human Interaction, 31(3), 1-21.
  Terfurth, Leonie; Gramann, Klaus & Gehrke, Lukas
  
• Exposing Movement Correlates of Presence Experience in Virtual Reality Using Parametric Maps. 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 374-380. IEEE.
  Gehrke, Lukas & Gramann, Klaus
  
• Modeling the Intent to Interact With VR Using Physiological Features. IEEE Transactions on Visualization and Computer Graphics, 30(8), 5893-5900.
  Nguyen, Willy; Gramann, Klaus & Gehrke, Lukas
  
• Visuo-haptic prediction errors: a multimodal dataset (EEG, motion) in BIDS format indexing mismatches in haptic interaction. Frontiers in Neuroergonomics, 5.
  Gehrke, Lukas; Terfurth, Leonie; Akman, Sezen & Gramann, Klaus
  
• Sense of Agency in Closed-Loop Muscle Stimulation. IEEE Access, 13, 105417-105433.
  Gehrke, Lukas; Terfurth, Leonie & Gramann, Klaus