Towards a neural network system architecture for multimodal explanations

• Applicant: Dr. Martin Dyrba
• Subject Area: Medical Informatics and Medical Bioinformatics ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 454834942

Advances in convolutional neural networks (CNN) have made it possible to achieve excellent diagnostic accuracy comparable to human raters for various disease detection tasks. However, CNN models are rarely applied in clinical studies or routine due to a lack of suitable methods for model comprehensibility and explainability. Thus, radiological assessments are based primarily on visual ratings of raw neuroimaging data and the professional experience of the (neuro)radiologist. Recent advances in methods for deriving explanations and visualizations from CNN models may help to overcome this limitation.The primary objective of this project is to investigate a system architecture and methodologies for deriving explanations from CNN models to reach a self-explanatory, human-comprehensible neural network. We will address the use case of detecting dementia and mild cognitive impairment due to Alzheimer’s disease (AD) and frontotemporal lobar degeneration in magnetic resonance imaging (MRI) data.As a first step, using a participatory design approach, we will examine which diagnostic criteria are applied by neuroradiologists and which expectations they have on software for diagnostic assistance with respect to comprehensibility and explainability. In a second step, based on our existing CNN model to accurately detect AD in T1-weighted brain MRI scans, we will develop a CNN system architecture capable of learning visual and semantic concepts that encode image features of high diagnostic relevance. Here, the development of a methodology for providing visual and textual explanations, including the definition of individual components and interaction of system components is a major challenge. We aim to create an attention model that is capable of a hierarchical representation of detected concepts, their diagnostic relevance as well as the ranking, selection and combination of potential explanations. In a third step, we will develop a visual interface for providing visualizations, explanations and user interaction methods. Here, interactive CNN relevance maps will show the brain areas with highest contribution to the diagnostic decision based on a single person’s MRI scan. We will create a textual summary elaborating the CNN decision process and detected image concepts of diagnostic relevance. Users shall be able to obtain more detailed information on specific image areas or individual explanation sentences interactively. Finally, the prototype CNN model will be evaluated by neuroradiologists with respect to its effect on diagnostic accuracy and the radiological examination process. Hereby, we will obtain information on the system’s utility and whether it actually improves the neuroradiologists’ diagnostic confidence and reduces examination time.With this project, we will develop and provide a methodology for easy-to-use and self-explaining CNN models. Additionally, we will provide a framework for generating clinically useful explanations.

• DFG Programme: Research Grants