Projection-Based Ultra Wide-Angle and 360° Video Coding

• Applicant: Professor Dr.-Ing. André Kaup
• Subject Area: Communication Technology and Networks, High-Frequency Technology and Photonic Systems, Signal Processing and Machine Learning for Information Technology
• Term: from 2019 to 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 418866191

Both surveillance and automotive applications make use of ultra wide-angle cameras to capture a field of view of more than 180 degrees. With 360-degree cameras, a horizontal field of view of 360 degrees is even possible. Such sequences offer a complete surround view of the captured scene and are thus of particular interest in the context of virtual reality applications, where they provide an immersive experience. In order to provide a sufficient visual quality, a resolution of the single video frames of 4K or 8K is used, which leads to an immense amount of data that cannot be transmitted or stored without compression. Conventional coding techniques do not consider the non-perspective projection functions of ultra wide-angle and 360-degree cameras, leading to suboptimal coding gains. The goal of this project is thus the development of novel compression techniques for ultra wide-angle and 360-degree video sequences to achieve an efficient storage and transmission. In a first phase of the project, a novel motion compensation technique has been developed that outperforms the the state-of-the-art projection-based approach and the traditional translational approach by 2.40 dB and almost 3.00 dB on average, a substantially improved handling of beyond 90-degrees incident angles has been derived, and a real-time interactive visualization tool demonstrating the developed algorithms has been established. In a second phase of the project, it shall be researched how and to what extend projection-based approaches can improve the compression efficiency of hybrid video coding and learning-based video compression network building upon the investigations from the first phase of the project. Thereby, the project addresses research on novel projection-based extensions to hybrid video coding, where special effort goes into the exploration of holistic approaches that take all dependent elements in the coding chain into account. Furthermore, due to the fast-paced development of promising end-to-end learned video compression techniques, the applicability of projection-based methods to end-to-end optimized video compression networks is researched aiming for improved compression efficiency of ultra wide-angle and 360-degree video. The generation and publication of suitable 360-degree datasets for training and testing of the developed techniques for hybrid and learning-based video coding is also part of the project.

• DFG Programme: Research Grants
• International Connection: Switzerland
• Co-Investigators: Dr.-Ing. Christian Josef Herglotz; Privatdozent Dr.-Ing. Jürgen Seiler
• Cooperation Partner: Professor Dr. Pascal Frossard