Factorization Approaches for 3D-Reconstruction of Rigid and Non-Rigid Scenes

• Applicant: Hanno Ackermann, Ph.D.
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2014 to 2018
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 250009097

It is one of the oldest problems in computer vision to estimate a 3D-reconstruction of a scene given a video sequence of an uncalibrated camera which generally moves. For a rigid scene and two, three or four images, it can be computed by epipolar geometry. For video sequences, a standard approach is to first estimate reconstructions by epipolar geometry using two, three or four images and use them to initialize a bundle adjustment. While such a standard approach exists for rigid scenes, there is no consensus within the scientific community about the mathematical model for objects which generally deform non-rigidly such as human faces.One such model is given by factorization approaches which are based on the decomposition of a data matrix in two matrices of lower rank. Factorization approaches have the advantage that all data are equally handled. They can be used for 3D-reconstruction of both rigid and non-rigid scenes, yet have the disadvantage that they suffer from missing 2D-correspondences.Two topics shall be addressed within this research project. Firstly, factorization approaches need be developed which can handle more than 70% missing data whereas state-of-the-art algorithms cease to be reliable if there are more than 40% unknown correspondences. To improve this threshold suitable image priors need be extracted and integrated into the optimization. The second focus is the 3D-reconstruction of generally deforming objects and scenes. Even recent algorithms suffer if only few neighboring points deform. To regularize the 3D-points image priors need be extracted. Furthermore, additional variables are going to be used for submatrices which induce a large model error.The aglorithms developed in both projects can be applied to other problems as well: Of special interest is customer recommendation which needs to consider missing data and changes of opinions over time - a particular form of non-rigid deformation.

• DFG Programme: Research Grants