ProPhys: Probabilistic Rigid-Body Physics Simulation for Vision-based Physical Scene Understanding

• Applicant: Professor Dr. Jörg-Dieter Stückler
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 466606396

Intelligent agents require the ability to understand their environment in order to act purposefully. A multitude of approaches for scene understanding from images have been proposed over the recent years that reconstruct geometry and detect objects. The representations learned by these methods lack a fundamental understanding of the physical properties of the environment on a Newtonian scale. Such physical understanding would enable intelligent robots to reason about their environment and to plan their actions. Physical scene properties and effects of actions could be displayed in virtual environments for novel applications of virtual reality and visualization.In this project, we will tackle the research question how physical scene understanding can be obtained by intelligent systems. We will investigate novel methods based on physical modeling, probabilistic reasoning and machine learning. We will research new methods for probabilistic physics simulation that include model uncertainty. The methods shall enable probabilistic prediction and reasoning to cope with noisy measurements, uncertainties in parameters and states, and to incorporate uncertain priors. Based on the simulation models, we will develop novel vision-based 3D scene understanding approaches that detect and track dynamic objects and estimate their physical properties. This way, more realistic physical scene representations can be obtained and tracking in challenging scenarios with occlusions or collisions can be achieved. We will also study the observability of physical properties from visual measurements and forces acting on objects and propose novel benchmarks for analysis and evaluation. This will guide future research and practitioners on the feasibility of computer-vision-based physical scene understanding.The proposed project has the potential to pave the way for new levels of scene understanding by intelligent systems. The methods developed in this project could enable intelligent robots to perform tasks in complex everyday environments more flexibly or open up novel applications in virtual and augmented reality.

• DFG Programme: Research Grants