Understanding and Creating Dynamic 3D Worlds towards Safer AI

• Applicant: Professor Dr. Carsten Rother
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2020
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 442613527

In the near future many AI-based systems will be deployed in real life scenarios. These are oftentimes dynamically changing environments where systems and humans have to safely co-exist. One important question is: Do AI-based systems operate reliably in all rare and critical scenarios? In this proposal we make a step towards answering this question. In particular we are building a system that actively generates rare and critical scenarios. To achieve this the project consist of two phases:a) building and learning a rich representation of dynamic 3D worlds based on visual inputs (images, videos, other sensors),b) smartly re-combining the different aspects of the rich 3D representation to generate rare and critical scenarios.The rich representation encompasses physical aspects (e.g. appearance, geometry, and dynamics) as well as semantic aspects (e.g. object class, meterial). Since this is a tremendous endeavor, we focus ourselves on a simplified AI-based system as a case study: Semantic understanding for outdoor driving scenarios with humans. The main goal is the generation of photorealistic, critical and rare scenes of dynamic environments. To achieve this, the technical goal is to push the state-of-the-art for rich 3D representations, including appearance, geometry, and dynamics, as well as semantic aspects. The main technical achievement is a software platform that enables other researchers to create critical and rare scenes for their application at hand.The collaboration involves three teams that have strong achievement records in the complementary aspects of the project:a) appearance modeling (Japan),b) dynamic 3D reconstruction (France),c) data generation and uncertainty analysis (Germany).In order to achieve the challenging goal of generating photorealistic dynamic scenarios, all partners are necessary and have to closely cooperate, especially towards the end of project. The project is a unique occasion to build an international network of complementary research centers with student and researcher exchanges to disseminate their respective expertise.We believe that the outcome of the project, as well as its international research network, will be at the forefront of computer vision based AI-systems. In the next 10 years we expect the world to be saturated with intelligent systems which are highly safety-critical (autonomous cars, robotic developments, to name a few). The ability to generate critical and rare scenarios to train and validate new systems will be essential to guarantee safety, and to further improve the existing systems.

• DFG Programme: Research Grants
• International Connection: France, Japan
• Partner Organisation: Agence Nationale de la Recherche / The French National Research Agency; Japan Science and Technology Agency; JST
• Co-Investigator: Professor Dr. Ullrich Köthe
• Cooperation Partners: Professor Dr. Ko Nishino; Dr. David Picard