Universal Continuous Semantic Mapping for Outdoor Environments

• Applicant: Professor Dr. Andreas Nüchter
• Subject Area: Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics
• Term: since 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 572721059

We propose the introduction of Universal Semantic Mapping, a comprehensive framework for outdoor semantic mapping. This framework will be designed to continuously capture and encode surface geometries and various surface properties (e.g., RGB color, laser reflectance, etc.) as well as higher-dimensional latent features (such as those found in CLIP embedding space). Semantic background knowledge is incorporated through the use of large language models. This will be the inaugural universal implicit encoding model capable of handling both geometry and an array of property types (such as RGB, infrared, or latent features) without the necessity of prior training augmented by Large Language Models (LLMs). The Uni-Fusion method organizes the outdoor point cloud data into regular grid voxels, wherein each voxel contains a latent feature. This forms a latent implicit map (LIM), representing both geometry and a range of properties. As data is gathered incrementally during outdoor exploration, a local LIM is generated and merged into a global LIM, facilitating real-time incremental reconstruction of the environment. We address the challenges of outdoor environments, such as processing large-scale environments, handling lighting and appearance variations, and integrating semantics using LLMs for adding background knowledge. Furthermore, we will advance LIM computing and explore applications.

• DFG Programme: Research Grants