Servicenavigation

Project Details

Back

Projekt Print View

Artificial Intelligence and Machine Learning in Minimally Invasive Upper GI Cancer Surgery: Workflow Analysis and Intraoperative Event Prediction towards Risk Mitigation

Subject Area General and Visceral Surgery
Term from 2023 to 2024
Project identifier Deutsche Forschungsgemeinschaft (DFG) - Project number 520324645
 
Final Report Year 2025

Final Report Abstract

This project aimed to investigate how artificial intelligence (AI) and machine learning (ML) can be used to analyze aspects related to surgical safety and outcomes in robotic-assisted minimally invasive esophagectomy (RAMIE) for upper gastrointestinal cancers. It was structured around three specific aims: (1) developing a standardized data and annotation framework suitable for RAMIE videos at the University Hospital Cologne, (2) modeling surgical workflows using AI-based phase detection and action recognition, and (3) analyzing fluorescence imaging (under Indocyanine Green (ICG) application) through computer vision (CV) to support dissection guidance. First, the project successfully established a comprehensive video data management and processing framework for RAMIE video data and associated metadata. The framework was informed by a Delphi consensus among international experts and resulted in a widely cited publication on data management (1). Based on this framework an automated recording and video processing manual was established at the University Hospital Cologne to ensure the future scalability of the video management pipeline across multiple sites and data modalities. For this purpose the institutional implementation of the Surgical Safe Technology Black Box System was initiated to ensure synchronized capture of video, environmental, and physiological data. Based on the continuously growing dataset (01/2025: n=196 RAMIE videos, abdominal and thoracic part, clinical metadata) we defined visual features to be annotated, including outcome critical characteristics and procedural phases and steps with particular focus on phases critical to surgical outcomes such as the ICG check after gastric conduit preparation and the anastomotic reconstruction phase. Subsequently, for Specific Aim 2 we developed complex ML model architectures, suitable for temporal and spatial analysis of lengthy, complex surgical procedures with no linear workflow and complex dependencies, such as RAMIE. Drawing from publicly available datasets on laparoscopic cholecystectomy videos we finetuned the algorithms to be specific yet generalizable enough to be suitable for the RAMIE use cases defined by the previously developed annotation guideline. Computationally, Transformer-based and Hypergraph-based (2) model architectures constitute very novel approaches to analyze surgical imaging data, yet we have demonstrated successful application to the selected datasets, which lead us to training these algorithms on the rich and complex RAMIE dataset with the aim of detecting and predicting surgical tool use, actions, and tissue interactions. In parallel the project explored the integration of ICG-based near-infrared (NIR) imaging into a CV framework to support intraoperative guidance. The ESOMAP study, which had started at the beginning of the funding period, confirmed the feasibility of fluorescence-guided lymphatic mapping in RAMIE. The developed Concept Graph Neural Networks(3) and Hypergraph architectures are currently being explored to model the complex, multi-relational data of the ESOMAP study(4), with the aim of analyzing temporal fluorescence intensity patterns and spatial distribution during critical steps to allow for ML based quantification of ICG dynamics. Moreover, in the future correlation with intraoperative histopathology (Stimulated Raman pathology) is planned. Overall, this project achieved most of its predefined goals and laid essential groundwork for future AI-based clinical decision support systems in oncologic upper gastrointestinal surgery. It established reproducible data pipelines, validated advanced modeling techniques, and fostered collaborative networks across institutions. The resulting publications and findings have already influenced the surgical AI community and will support future clinical translation.

Publications

 
 

Additional Information

© 2026 DFG Disclaimer / Copyright / Privacy Policy

Textvergrößerung und Kontrastanpassung