Polymorphic Scene Representation for Enhanced Instant Scene Reconstruction

• Applicant: Professor Dr.-Ing. Andreas Kolb
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 510825780

The main goal of real-time 3D scene reconstruction is to accurately acquire and represent the geometry, as well as other properties of the scene being captured, such as appearance information, decomposition into shape primitives, and scene dynamics. While robust real-time reconstruction of 3D scene geometry has seen significant improvements, there are still substantial deficiencies in high-quality real-time acquisition of the other scene properties. The core hypothesis of this project is that the primary cause of these deficiencies is inherent in the shortcomings of the particular representation type (usually point sets or voxel grids) used to depict the scene information. Therefore, the goal of the project is to develop a polymorphic scene representation for improved real-time reconstruction of scene properties, incorporating extensive use of different representation types (points and voxels, but also polygon meshes, images and novel extended types), as well as efficient polymorphic operators on the polymorphic scene representation that hide the locally used representation type. According to the core hypothesis, the polymorphic scene representation and operators enable seamless integration, as well as fundamentally enhanced progressive real-time reconstruction of the various scene information, \ie, geometry, appearance, shape composition, and scene dynamics. The development and use of a polymorphic scene representation and corresponding operators requires, inter alia, efficient GPU-based processing and storage techniques, the inclusion of previously underutilized representation types such as polygon meshes and images, and new concepts for efficient fusion and abstraction of scene information. The study of polymorphic scene representations that use multiple representation types will lead to an improved theoretical understanding of the complementary nature of the basic representation types and the potential of their parallel utilization to leverage their individual strengths. In addition, the practical availability of polymorphic scene representations will improve the use of real-time 3D scene reconstruction techniques due to their ability to adapt to specific, spatio-temporal requirements, thus providing new momentum in application fields beyond real-time 3D scene reconstruction, such as in 3D scene analysis or shape similarity of deformable objects.

• DFG Programme: Research Grants