5D Adaptive Light Field Sampling (FiDALiS)

• Applicant: Professor Dr.-Ing. Thorsten Herfet
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2019 to 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 429078454

• Final Report Year: 2024

Final Report Abstract

The primary achievements of the FiDALiS project can be summarized as follows: We introduced the Froxel representation as a method for presenting light fields in a ray-centric manner. This was achieved by discretizing the view frustum of a light field array into Froxels, which were sized in accordance with the resolution of the array. 2 Suitable datasets composed of both real-world and synthetic data were collected. To underscore the limitations of current sampling strategies, we enhanced the Blender Classroom scene with moving objects. We proposed an efficient method for storing and processing sparse Froxel data using hash maps. Additionally, we developed a GPU implementation to demonstrate the extensive parallelizability of our approach. A depth processing technique, based on Froxels, has been proposed that enables the generation of high-quality Froxel representations, even in the absence of ground truth depth maps. For instance, depth maps were extracted from a NeRF to highlight the symbiotic relationship between novel neural approaches and the proposed Froxel representation. The Froxel representation is a powerful tool that can introduce semantics into light field data and is capable of scaling to light field videos. It enables analysis in the spatial (Fristograms), visual (surface classification), and temporal (spatio-temporal sampling distribution) domains. The developed software and the generated dataset are publicly available on Zenodo. The results outlined above have led to multiple publications in organs with scientific quality assurance, highlighting the successful management of challenges posed by SARS-CoV-2. Industry roles of the inital lead researcher within the field of visual information processing resulted ultimately in the successful acquisition of joint industry funding from Saarland and Intel Corporation under the program ”The Future of Graphics and Media”. The goal of the accepted proposal ”FROST: FROxel-based Semantic Processing Techniques” is to further develop the Froxel representation and develop real-world applications on top of it.

Publications

• Deep Learning-based Semantic Analysis of Sparse Light Field Ray Sets. 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), 1-6. IEEE.
  Chelli, Kelvin; Tamboli, Roopak R. & Herfet, Thorsten
  
• Fristograms: Revealing and exploiting light field internals
  T. Herfet, K. Chelli, T. Lange & R. Kremer
  
• Gaining insights into the information distribution of light fields and enabling adaptive light field processing. SKILL 2022
  R. Kremer
  
• Acquisition of light field images & videos. Immersive Video Technologies, 163-171. Elsevier.
  Herfet, Thorsten; Chelli, Kelvin & Le, Pendu Mikael
  
• Light field representation. Immersive Video Technologies, 173-199. Elsevier.
  Herfet, Thorsten; Chelli, Kelvin & Le, Pendu Mikael
  
• SAIL: Semantic Analysis of Information in Light Fields: Results from Synthetic and Real-World Data. Computer Science Research Notes, 90-99. University of West Bohemia, Czech Republic.
  Kremer, Robin & Herfet, Thorsten
  
• ST-SAIL: Spatio-temporal Semantic Analysis of Light Fields: Optimizing the Sampling Pattern of Light Field Arrays. 2024 IEEE International Conference on Consumer Electronics (ICCE), 1-6. IEEE.
  Kremer, Robin & Herfet, Thorsten