Deep Models for Handheld Light Field Acquisition

• Applicant: Dr. Paramanand Chandramouli
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2020
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 437172262

Image-based rendering (IBR) is a central and interdisciplinary research topic in computer graphics and computer vision. It comprises the acquisition, representation and synthesis of light fields that are mainly based on a set of previously acquired images, \ie light field samples. Depending on the approach, additional depth data is used for efficient image interpolation. Even though investigated for more than 20 years, the acquisition of high quality light field data commonly undergoes controlled restrictions with regard to the acquired parameter sub-space and/or dimensions of the underlying, high-dimensional plenoptic function. Therefore, the acquisition of high quality light field data is either restricted to low angular or spatial variations (small baselines) when using, \eg, plenoptic cameras or it requires sophisticated and exhaustive hardware setups comprising dozens of cameras. Recently, learning based approaches have also been used in light field research, mainly addressing plenoptic cameras, \ie light fields with small baselines.This research project aims at a lightweight approach for the online acquisition of wide baseline light fields for unrestricted scene configurations using a single RGB-D camera and incorporating deep learning models. It follows two main research directions that are eventually integrated to achieve this goal. The first research direction addresses the development of novel deep learning models that efficiently represent small baseline light fields in a generic way. They will allow for efficient light field storage, processing and reconstruction under different, mainly sparse observation models. The second research direction focusses on the progressive reconstruction for 3D surface light fields for arbitrary scene topology. This line of investigation requires novel approaches to scene representation, camera pose estimation and data accumulation. The main hypothesis of integrating these lines of research is that the deep models will be able to reconstruct local light fields for arbitrary camera motion. This, in turn, will be the fundamental basis for highly accurate camera pose estimation and correspondence finding, that is necessary to achieve progressive online, high quality dense light field reconstruction even for complex reflection scenarios.

• DFG Programme: Research Grants