SEASONAL - Smart Estimation and Alteration of Scenes in Outdoor and Nature Areas using machine Learning

• Applicant: Professor Dr. Wolfgang Broll
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 535507286

The goal of SEASONAL is the deep learning-based transformation of outdoor images between different seasons, weather, and other environmental conditions. Transformations from one visual domain to another, also known as image-to-image translation, form the basis for various image processing tasks such as super-resolution, image colorization, or inpainting. Moreover, beyond qualitative improvements, image-to-image translation can be used to modify the semantic content of an image. However, existing approaches often consider only pairwise relations between individual domains and require image pairs for the training process. Also, the evaluation usually focuses only on the quality of the generated images and does not consider their plausibility. In this project, we generate images that are visually pleasing but also highly plausible images while handling the transformation between multiple domains. This includes not only a conversion between static domains such as seasons (e.g., summer to winter) but also an individual adaptation to a wide range of weather and lighting conditions. A challenge in this context is creating a suitable and comprehensive training data set that adequately covers such conditions in various outdoor scenarios. In addition, we design a neural network architecture to enable the multi-domain translation of images with respect to different environmental features. Furthermore, we explore methods to increase the quality and the plausibility of the generated images. We then evaluate the results with a combination of objective metrics (e.g., SSIM, FID) and user studies. Thus, this project immediately contributes to the research area of generative models, especially with regard to image-to-image translation, and in the long term, acts as a core building block for future applications in the areas of automatic content-based image manipulation and enhancement.

• DFG Programme: Research Grants