Advanced Image Sensing Using Arbitrarily Shaped Pixels and Neural Network Reconstruction

• Applicant: Professor Dr.-Ing. André Kaup
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Communication Technology and Networks, High-Frequency Technology and Photonic Systems, Signal Processing and Machine Learning for Information Technology
• Term: since 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 516695992

Objective of this project is the systematic conception of novel image sensors with non-regularly shaped pixels and the design of neural network-based image reconstruction techniques. Thereby the image quality shall be significantly improved compared to regular square pixels in combination with state-of-the-art single-image super-resolution algorithms while retaining the same number of samples. Non-regular sampling has advantages over regular sampling as it can result in higher resolution per sampled pixel. An application in image processing is a sensor concept called 1/4 sampling. In 1/4 sampling, each square pixel of an image sensor is covered such that only one randomly chosen quadrant is left transparent. Since the fill factor is decreased by a factor of four that way, and 75 % of the incoming light is lost in such an implementation, solutions were searched to keep the non-regularity while increasing the fill factor. One promising way that is investigated in this project is the usage of non-regularly shaped pixels covering the full sensor area at 100 % fill factor. There are several tilings of areas available that are promising for this task. In contrast to a regular sensor with square pixels, the measured data needs to be processed to reconstruct the image on a regular grid. While a model-based approach for similar tasks is known, we are planning to expand this to the usage of neural networks as these have shown promising results in related fields of research. The use of neural networks for non-regular sampling has been poorly investigated and is promising. Such, the new sensor layouts combined with new neural network-based reconstruction methods have great potential for novel higher resolution camera sensors.

• DFG Programme: Research Grants