Automated real-time image fusion of radiological image data and patient anatomy in augmented reality using artificial intelligence

• Applicant: Dr. Malte Sieren
• Subject Area: Radiology ; Medical Informatics and Medical Bioinformatics
• Term: since 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 519070549

The project aims to develop an AI-supported, automated image fusion process/pipeline of 3D computed tomography (CT) image data with patient anatomy in augmented reality to support surgical and interventional procedures. This aim will be achieved with the help of the camera systems integrated into the AR glasses and a 3D Spatial Transformer Network. The cameras capture the patient anatomy in real-time, and an AI algorithm then automatically fuses the 3D data reconstructed from the CT dataset with the anatomy based on anatomical landmarks. The fusion result is permanently re-evaluated in real-time based on an AI algorithm, and, if necessary, adjusted accordingly. As a use case, the application in endovascular interventions for the targeted puncture of the common femoral artery in a patient study will be demonstrated. The focus is not on automating tasks previously performed by interventionists but on synergistically supporting human expertise through technology. The application of AI promises to facilitate the use of AR and would theoretically also be transferable to other AR applications in the medical field. The successful implementation of the clinical use case could positively impact the success rates of endovascular interventions and patient outcomes. Furthermore, it can be assumed that the already scientifically proven advantages of conventional fusion imaging (e.g., dose savings) will be further augmented.

• DFG Programme: Research Grants
• Co-Investigator: Professor Dr. Floris Ernst