Visual-Haptic Perceptual Illusions for Hand-based Interaction in Virtual Environments

• Applicants: Professor Dr. Antonio Krüger; Professor Dr. Frank Steinicke
• Subject Area: Human Factors, Ergonomics, Human-Machine Systems ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 450247716

Virtual Reality (VR) enables users to experience computer-generated environments and digital data in a natural and immersive way, in particular, by allowing users to directly grab virtual objects with their hands. Unfortunately, most current VR experiences still lack realistic haptic feedback, which hinders that users can truly ‘feel’ the virtual environment (VE) including tactile, kinesthetic, proprioceptive and vestibular cues. However, during hand-based interactions, users do not only rely on haptic and tactile feedback, but instead multisensory cues from vision, feeling and hearing are combined during the multisensory integration process. When senses are in conflict, the human brain weights different stimuli according to their reliability. Since visual perception often dominates during this integration process, visual illusions can be used in order to alter the haptic perception, which is often referred to as crossmodal interactions. Such crossmodal interactions for hand-based VR interaction using active and passive haptics have the potential to provide a compelling high-fidelity multimodal feedback. However, basic research is required to fully understand the crossmodal interplay between vision and haptics for human-computer interaction (HCI) in VR.In this proposed research project, we aim to investigate the interplay between vision and haptics to provide novel forms of visual-haptic hand-based VR interaction. First, we aim to investigate the effects of visual dominance and perceptual illusions to alter interaction with virtual objects. For this, we study techniques that actively modify how the user visually perceives the VE. For example, with redirected touching the user's virtual hands are displaced with an offset from the real hand positions to modify the user's movement towards or across physical proxy surfaces. In the second phase, we focus on how passive proxy design and active feedback can enhance haptic-based VR interactions. We plan to explore the space of visual-haptic feedback for VR spanned by the active-passive haptics continuum and an active-passive visual feedback dimension. In this context, we aim to investigate how actuators, active vibration feedback, passive haptic props and specific visualizations can enhance interaction and perception in VEs. Finally, we will provide an open-source framework including manufacturing schematics and instructions for the audience interested in building their own VEs supporting visual-haptic perceptual illusions.

• DFG Programme: Research Grants