Rolling Diffusion World Models for 3D Environment Perception and Action Planning in Intelligent Systems

• Applicant: Professor Dr. Jörg-Dieter Stückler
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 466606396

Autonomous robots need an understanding of the effect of their actions for planning and executing tasks in the environment. In this project, we will devise novel methods for learning-based physics simulation and 3D perception of the environment. We will develop learning approaches for world models that allow for probabilistic, multimodal prediction of object motion in response to interactions. Based on these world models, we will investigate methods that reconstruct the environment in 3D from RGB-D images and use the environment model for planning of actions. The approaches will be evaluated for 3D perception of dynamic scenes and object manipulation by autonomous robots in virtual and real environments. We will develop novel approaches based on diffusion models that support probabilistic, multi-modal dynamics prediction. The environment will be represented in volumetric radiance fields using dynamic 3D Gaussians. For efficient computation for generation, perception, and planning, we will develop methods for prediction in rolling time windows and with multi-resolution in time. We anticipate that such world models will be useful for scene understanding by virtual assistants for foreseeing and proposing possible actions. In robotics, world models could be used to flexibly and adaptively gain understanding of dynamical properties of environments and use this information for goal-directed action planning. The approaches developed in this project have large potential for enabling future applications of virtual assistants and robotics in real unstructured environments such as the home, logistics, or the crafting sector. Moreover, our research could inspire novel research in related disciplines such as neuro and cognitive science.

• DFG Programme: Research Grants