Robust Optical Flow

• Applicant: Professor Dr.-Ing. Andrés Bruhn
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 533085500

In computer vision, the steadily growing demand for motion estimation algorithms in real-world applications such as autonomous driving or medical imaging renders the design of robust optical flow methods a fundamental aspect of future research. So far, however, in the development of optical flow methods, robustness clearly stands back behind aspects such as quality or run time, which are frequently evaluated and have improved drastically due to the progress in the field of neural networks. Moreover, the few existing approaches to quantify and improve robustness have hardly sparked the development of more robust methods so far, which is also substantiated by recent findings that current state-of-the-art-methods are particularly vulnerable. Hence the main goal of this project is to promote and advance the development of robust optical flow methods. This shall be achieved by providing novel and easy-to-use frameworks for a rigorous quantification of robustness that allows to improve the robustness of optical flow methods by advancing the model design and crafting training data. Overall, this enables the creation of optical flow methods with an improved robustness that retain a good qualitative performance at benchmarks. Concept-wise, to quantify robustness, the projects considers the robustness towards out-of-distribution data and adversarial attacks. Both concepts have widely been used to quantify the robustness of classification networks, and both image corruptions and adversarial attacks were recently shown to affect optical flow networks. While out-of-distribution robustness assesses the generalization capabilities of a model, adversarial attacks consider worst-case scenarios and offer a mathematically motivated robustness definition. Due to their complementary nature, both robustness concepts are appealing tools to analyze the robustness of optical flow methods. This has not only the potential to advance the robustness of optical flow methods, but also to improve the understanding of the different types of robustness are interconnected in the context of optical flow. The development of novel more robust optical flow methods will proceed in four steps. In a first step, novel analysis tools for out-of-distribution robustness and adversarial robustness for optical flow shall be developed that allow a more meaningful quantification of robustness in realistic and worst-case scenarios. In a second step, these evaluation tools enable us to analyze optical flow methods on a design-level, in order to identify structures (architectures, layers, losses) that enhance robustness. Similarly, in a third step, the analysis is extended to the data-level, where suitable datasets, augmentations and training strategies are built for an improved robustness. Finally, in a fourth step, the insights of this systematic analysis shall be fused into a novel robust optical flow method, and be used to create general guidelines for building robust models.

• DFG Programme: Research Grants