BodyLie - SLAM

• Applicant: Dr. Nicolas Mandel, Ph.D.
• Subject Area: Human Factors, Ergonomics, Human-Machine Systems ; Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics
• Term: since 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 565664207

3D positioning of humans in space from monocular images is a fundamental problem that has many applications ranging from human-robot interaction to ergonomics and biomechanical analysis. Although learning models have made tremendous progress in this area and perform on par with commercial tracking systems in laboratory environments, the performance in the real world so far falls short of expectations. The aim of the project is to create and implement a grey-box model that combines the advantages of learning methods and physical models. For this purpose, a 3D motion model of the human body is combined with a camera model in order to link the 2D projection of the body in the camera images with static landmarks and create a 3D reconstruction. The foundation is a Simultaneous Localisation and Mapping (SLAM) formulation in the form of a factor graph. The underlying mathematics, Lie groups, are used to model human body motion. The result is a representation of the body and camera in 3D over time that incorporates the uncertainties of the model. The integration of the uncertainty allows introspective conclusions to be drawn about the properties of the model. The project will investigate how sensitive the model is to influences of movement and appearance and which open problems of 3D reconstruction can be solved by the additional information. Furthermore, the impact of the generated map representation on downstream analyses such as event classification, kinematic analyses or biomedical assessments will be investigated. The key aims of the project are: • Developing a physically consistent model using Lie group maths that integrates human body motion together with camera motion and observation generation • Identifying or collecting a validation dataset with 3D ground truths for human pose and camera. • Developing a software package that integrates 3D human motion, 2D recognition models and factor graph optimisation. • Performing sensitivity analysis to identify robustness issues and the potential for solving open problems in 3D reconstruction. • Investigating the use of representation with uncertainty as an embedding space for action recognition. • Exploring the potential to feed information from factor graph optimisation back to the detector. This project aims to develop a robust and generalisable solution that makes 3D human pose detection as ubiquitous as SLAM. It aims to enable the application of these models in environments where reasoning about human poses is crucial, such as in biomechanical analysis or human-robot interaction.

• DFG Programme: WBP Fellowship
• International Connection: USA
• Host: Professor Ehsan Adeli, Ph.D.