Mediated Reality using Radiance Fields (MURF)

• Applicant: Professor Dr. Wolfgang Broll
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Methods in Artificial Intelligence and Machine Learning
• Term: since 2026
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 575281243

Augmented and Mixed Reality enable the combination of virtual objects with the physical world, offering new ways to experience and interact with digital content. An advanced form of this technology is Mediated Reality --- a visual manipulation of reality that not only allows the addition of virtual objects but also the targeted removal of objects or the adjustment of photometric and geometric properties of the real environment. For these technologies to work seamlessly and convincingly, virtual objects must be correctly placed or removed within the physical scene, while also accounting for resulting changes in lighting, perspective, and material properties. The project MURF addresses these challenges by leveraging geometric and photometric reconstruction methods to enable novel visualization methods. The goal is to realistically add, modify, or remove objects and light sources in realtime. To achieve this, the project explores innovative approaches combining path tracing with 3D Gaussian Splatting, enabling high-quality and photorealistic rendering while maintaining efficient processing. The developed methods will undergo continuous technical evaluation and be validated through user studies on mobile hardware. The expected outcomes include the development of new algorithms and systems that will be cutting-edge for visualization and interaction in Mediated Reality applications. At the same time, these techniques will also be applicable in conventional graphics workflows and Virtual Reality environments.

• DFG Programme: Research Grants
• Co-Investigator: Dr.-Ing. Tobias Schwandt