Interpretable and Robust L2S - Optimization

• Applicant: Professorin Dr. Margret Keuper
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 459284860

Driven by the recent successes in machine learning, the goal of L2S is an end-to-end learning pipeline of image sensing and analysis systems which allows a downstream application-driven sensor design.Yet, while an end-to-end optimization is promising from a machine learning perspective, the complex interplay between input sensor data, neural network architecture and learned network weights amplifies several research questions that are to a certain extent, already prevalent in classical deep learning.First, it is to be expected that currently well performing architectures are somewhat tailored to the conventional RGB sensor design. Therefore, we argue that the sensor optimization should be accompanied by an optimization of the neural architecture along with the network weights. We will therefore address both, sensor and architecture optimization, in a joint discrete optimization problem. Bayesian Optimization over a joint architecture and sensor parameter search space can be employed to find optima in an efficient way. We will thereby explore different sensor layouts. For efficiency, we will consider architecture and sensor embeddings computed using graph neural networks.Second, the optimization with respect to improved accuracy on a downstream task such as image classification, semantic segmentation or optical flow computation, might yield architecture/sensor pairs that perform very well on the data available at training time. Yet, the robustness of such models even under slight domain shifts is expected to be more brittle than in conventional learning driven approaches, where the sensor is not subject to the end-to-end optimization. Therefore, we will investigate methods to ensure a certain robustness of the resulting model/sensor pairs using for example regularization and data augmentation techniques.Third, while neural networks often issue highly accurate predictions, their interpretability is usually low and one has very limited understanding of the prediction uncertainty. Both issues are reinforced when addressing a joint sensor/machine learning optimization. On the one hand, we will investigate how existing methods for network visualization, that help to make the decision process more interpretable, can be transferred to optimized sensors that do not necessarily output image-like data. On the other hand, we will analyze how locally optimized sampling on the sensor side interferes with decision uncertainties, measured for example using Monte-Carlo drop-out. All questions will be considered in the context of sensor modalities ranging from optimized RGB sensors over terahertz measurements to lightfield microscopic recordings and with respect to applications that require varying localization precision, such as classification (no localization), segmentation (pixel accurate localization) or optical flow estimation (pixel accurate localizaiton, sub-pixel accurate mapping).

• DFG Programme: Research Units
• Subproject of: FOR 5336: KI-FOR Learning to Sense