Image Synthesis as an Epistemic Method Towards Understanding Art

• Applicants: Professor Dr. Peter Bell; Professor Dr. Björn Ommer
• Subject Area: Art History ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2019 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 421703927

A great advantage of digital images is the potential to bring large numbers of artifacts virtually together to then easily link them to related samples, flexibly rearrange, or to simply order them in database systems. Much like in Aby Warburg’s mnemosyne atlas, digital images are therefore constantly brought into relation to another. However, relations and similarities or dissimilarities between large numbers of art works are based on potentially fairly abstract concepts. Especially when computers establish such relations, e.g., retrieval results. Therefore, Computer Vision has recently shown great interest in a special kind of deep learning approaches that are generative in nature. Such methods allow to directly visualize the abstract representations that they learn. This proposal will work with neural networks that synthetically generate digital (SGD) images to explain the representations they have learned for art collections. These SGD images establish a new means of access to concepts in collections of digital or digitized art by distillation. Consequently, our goal in this project is to challenge the way art history views the digital image. The digital image should convert towards an epistemic instrument. Rather than only being the object of an art historical analysis, we will empower SGD images to become a valuable means for the process of analysis. The project tackles the hermeneutic questions of reading not only a 'computer generated image' but the underlying manifold.We will develop a generative deep learning approach which allows to SGD images that visualize the representations learned by existing or new algorithms for image retrieval and analysis. We explicitly disentangle different concepts of a set of images. Not only can our approach then visualize these concepts, such as different artistic styles and different poses of a human figure. It also enables a user to validate an individual detection, by seeing what the underlying retrieval algorithm has captured from the image. In contrast to existing network inversion techniques without explicit disentangling, the user can now alter the different concepts individually and study if they have been correctly represented and separated. As a case study we will explore SGD images of antiquity to generate artificial renaissances to evaluate the epistic capacity of the neural networks.Finally this enables an interactive approach for learning and analysis of concepts from large sets of images. Erroneous detections can be easily spotted in large numbers of images without having to evaluate them one at a time. Thus computer and art historian can explore image collections in an alternating manner. The computer presents distillations of the concepts that have been discovered so far and the art historian guides the further analysis and interprets the resulting images. According to that we use the analytic competence concerning imagery from algorithms as well as art history and work on mixed methods.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2172: The digital Image