SIMSURGE: Balancing the odds by simulating rare cases for surgical data science

• Applicants: Professorin Dr.-Ing. Stefanie Speidel; Professor Dr. Martin Wagner
• Subject Area: General and Visceral Surgery ; Methods in Artificial Intelligence and Machine Learning ; Medical Informatics and Medical Bioinformatics
• Term: since 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 560101272

Modern machine learning methods require vast amounts of data to train. In Surgical Data Science, this gives rise to several challenges, because there is not only much less data available overall, but encountered situations are also extremely heterogeneous - due to every patient, disease, surgical procedure and hospital bearing unique characteristics. Moreover, especially challenging cases and complications occur rarely. Ideally, the data used to train machine learning algorithms, robots and also young surgeons should reflect this diversity, cover edge-cases in sufficient quantity and represent all patients fairly, ultimately aiding in the creation of trustworthy assistance systems. Computer simulations can offer a way to increase the data amount and are becoming a valid data source for training both computer systems (reinforcement learning) and novice surgeons (minimally invasive training simulators). However, these simulations often cover only basic, clean and common situations rather than featuring diverse content, since designing edge-cases often requires too much engineering. Within the SIMSURGE project, we will explore semi-automatic methods to generate vast amounts of diverse, realistic surgical simulations of rare surgical situations (Fig.1). They will be based on physical simulation, which ensures that they remain controllable and ground-truth labels are available for every scene and image, and enriched via large vision language models which will add diversification, expand the number of cases that can be simulated and ensure that the generated datasets are balanced. We will evaluate the system by performing studies related to data-bias, by training reinforcement learning algorithms for automation tasks as well as user studies in which we validate the data with surgeons and medical students in various levels of expertise. The project tackles multiple unsolved research questions in surgical simulation, translation and data curation, making it a high-risk undertaking (see also “Feasibility” below). However, the benefits which the described low-cost high-fidelity synthetic data would bring to the areas of computer-assisted intervention, surgical robotics and surgical training would be enormous. To facilitate such usage and increase impact, all developed methods and the generated synthetic datasets will be made available to the public after a thorough validation process.

• DFG Programme: Reinhart Koselleck Projects