Losless and lossy compression of screen-content data using machine learning

• Applicants: Professor Dr.-Ing. André Kaup; Professor Dr.-Ing. Tilo Strutz
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2020 to 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 438221930

• Final Report Year: 2024

Final Report Abstract

These days, the use of applications such as online conferencing, e-learning or display sharing rises. Especially during the corona pandemic starting 2020, many universities and other learning facilities started to incorporate e-learning into their mode of operation. For such applications, screen content (SC) data, i.e., video and image data generated directly by the computer or smart phone, has to be transmitted or saved. In contrast to sensor-generated data, SC usually consists of text, buttons, icons and rendered-graphics. As a results, SC has diverse characteristics which strongly differ from sensor-generated data. Often, SC contains only few unique colors and patterns, sharp contrasts, fine features and areas of uniform color. Consequently, image and video codecs developed for sensor-generated content perform poorly on SC data. Instead, it has been shown that a method based on ideal entropy coding and modeling of probability distributions for each pixel shows excellent compression performance on SC data with a small number of patterns and colors. However, the prototype is restricted to lossless compression and its efficiency decreases when the SC data has certain characteristics. In this project, we improve the efficiency of the prototype for SC data which contains natural image areas by introducing a segmentation, which allows the codec to learn separate statistics for natural and synthetic image areas in compound SC images. Improved probability modeling for prediction error statistics and color palette statistics further enhances the compression efficiency of the prototype for SC data. Additionally, the prototype is incorporated into Versatile Video Coding (VVC) as a novel coding tool. We show that video compression and mixed lossy and lossless compression using the prototype compression method as a coding tool outperform other state-of-the-art methods.

Publications

• Optimization of Probability Distributions for Residual Coding of Screen Content. 2021 International Conference on Visual Communications and Image Processing (VCIP), 1-5. IEEE.
  Och, Hannah; Strutz, Tilo & Kaup, Andre
  
• Image Segmentation for Improved Lossless Screen Content Compression. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1-5. IEEE.
  Uddehal, Shabhrish Reddy; Strutz, Tilo; Och, Hannah & Kaup, André
  
• Rescaling of Symbol Counts for Adaptive rANS Coding. 2023 31st European Signal Processing Conference (EUSIPCO), 585-589. IEEE.
  Strutz, Tilo
  
• Enhanced Color Palette Modeling For Lossless Screen Content Compression. ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3670-3674. IEEE.
  Och, Hannah; Uddehal, Shabhrish Reddy; Strutz, Tilo & Kaup, André
  
• Improved Screen Content Coding in VVC Using Soft Context Formation. ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3685-3689. IEEE.
  Och, Hannah; Uddehal, Shabhrish Reddy; Strutz, Tilo & Kaup, André