Immersive Digital Reality

• Applicant: Professor Dr.-Ing. Marcus Magnor
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2016 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 283369270

• Final Report Year: 2024

Final Report Abstract

Over the course of the Reinhart Koselleck project “Digital Immersive Reality”, we developed solutions for different challenges encountered towards the goal of being able to watch real-world videos immersively and turning immersive entertainment into a social experience that can be enjoyed in a group of people. Particularly, the project contributed viable methods to • omnidirectional capture of dynamic, real-world content, • free-viewpoint render from multi-viewpoint panorama recordings as well as from omnidirectional stereo videos, • guide visual attention in immersive displays, • mitigate cybersickness in immersive environments, • efficiently compress and real-time decode very high-resolution omnidirectional videos on graphics hardware, • facilitate instant re-direction in VR based on muscle stimulation-triggered eye blinks, and • create realistic digital doubles of real people, animation-controlled by simple video. Looking beyond the finished Reinhart Koselleck project, our research continues to aim at importing realworld recordings into immersive environments. Goal of the new project ”Real Action VR” is to utilize action cams to create immersive live-action experiences of our own or other people’s adventures.

Publications

• Video Based Reconstruction of 3D People Models. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8387-8397.
  Alldieck, Thiemo; Magnor, Marcus; Xu, Weipeng; Theobalt, Christian & Pons-Moll, Gerard
  
• Learning to Reconstruct People in Clothing From a Single RGB Camera. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1175-1186.
  Alldieck, Thiemo; Magnor, Marcus; Bhatnagar, Bharat Lal; Theobalt, Christian & Pons-Moll, Gerard
  
• Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama Video. 26th ACM Symposium on Virtual Reality Software and Technology, 1-9.
  Mühlhausen, Moritz; Kappel, Moritz; Kassubeck, Marc; Bittner, Paul M.; Castillo, Susana & Magnor, Marcus
  
• High-Fidelity Neural Human Motion Transfer from Monocular Video. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1541-1550.
  Kappel, Moritz; Golyanik, Vladislav; Elgharib, Mohamed; Henningson, Jann-Ole; Seidel, Hans-Peter; Castillo, Susana; Theobalt, Christian & Magnor, Marcus
  
• Towards Understanding Perceptual Differences between Genuine and Face-Swapped Videos. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1-13.
  Wöhler, Leslie; Zembaty, Martin; Castillo, Susana & Magnor, Marcus
  
• Visual Techniques to Reduce Cybersickness in Virtual Reality. 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 486-487.
  Groth, Colin; Tauscher, Jan-Philipp; Heesen, Nikkel; Castillo, Susana & Magnor, Marcus
  
• Omnidirectional Galvanic Vestibular Stimulation in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics, 28(5), 2234-2244.
  Groth, Colin; Tauscher, Jan-Philipp; Heesen, Nikkel; Hattenbach, Max; Castillo, Susana & Magnor, Marcus
  
• Immersive Free‐Viewpoint Panorama Rendering from Omnidirectional Stereo Video. Computer Graphics Forum, 42(6).
  Mühlhausen, Moritz; Kappel, Moritz; Kassubeck, Marc; Wöhler, Leslie; Grogorick, Steve; Castillo, Susana; Eisemann, Martin & Magnor, Marcus
  
• Instant Hand Redirection in Virtual Reality Through Electrical Muscle Stimulation-Triggered Eye Blinks. 29th ACM Symposium on Virtual Reality Software and Technology, 1-11.
  Groth, Colin; Scholz, Timon; Castillo, Susana; Tauscher, Jan-Philipp & Magnor, Marcus
  
• Wavelet-Based Fast Decoding of 360° Videos. IEEE Transactions on Visualization and Computer Graphics, 29(5), 2508-2516.
  Groth, Colin; Fricke, Sascha; Castillo, Susana & Magnor, Marcus