Camera Array for Hyperspectral Video Imaging Using Cross-Spectral Multi-View Fusion

• Applicant: Professor Dr.-Ing. André Kaup
• Subject Area: Communication Technology and Networks, High-Frequency Technology and Photonic Systems, Signal Processing and Machine Learning for Information Technology
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 491814627

Hyperspectral imaging records a sampled light spectrum for every pixel of a scene. There are a number of applications using hyperspectral imaging and video imaging, e.g. in agriculture, chemistry, and medicine. The challenge with hyperspectral imaging is that a three dimensional cube has to be recorded using two dimension sensors, since a grayscale image is recorded for every small wavelength area. Thus, this three dimensional image cube has to be unfolded. This can be done by using the temporal dimension, however, using this approach the ability of capturing videos is lost. Therefore, snapshot spectral imagers, which record the hyperspectral data cube in one shot, are highly desirable. One approach using off-the-shelf hardware for this is by usinga cross-spectral camera array. Here, each wavelength area is recorded by one camera, which is equipped with a suitable bandpass filter in front of the lens. Subsequently, a registration and reconstruction process is necessary to warp all camera views to one single view.The goal of this project is to setup a hyperspectral camera array using 37 channels, where the registration and reconstruction pipeline shall be entirely replaced by an end-to-end neural network. Using neural networks, a challenge is to create enough data for training, which will be tackled by building a hyperspectral renderer and generating synthetic hyperspectral sequences. As soon as the hardware and the end-to-end neural networks are set up and trained, a hyperspectral video database will be created and published, which is a novelty to the scientific community. Another challenge results from the fact that different cameras with varying physical properties are necessary when traversing different parts of the light spectrum. Consequently, the multi-device cross-spectral problem is tackled by adapting the end-to-end neural networks to support camera with different physical properties. Moreover, the camera arrangement with different camera-lens combinations is optimized.

• DFG Programme: Research Grants