Pose Estimation for Tracking Tomographic Microscopy

• Applicant: Professor Dr. Ivo Ihrke
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Optics, Quantum Optics and Physics of Atoms, Molecules and Plasmas
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 543691203

This proposal is concerned with the problem of 3D pose estimation for video sequences of non-uniformly rotating transparent or partially attenuating specimen in microscopic imaging. The ability to robustly extract pose from such videos would enable the tomographic 3D reconstruction of unstained living cells and cell clusters, potentially in a time-resolved manner, as is e.g. desired in cancer research for studying the development of emerging in-vitro tumor models and their possible treatment options. Application areas are the development of general cancer drugs and, potentially, in the future, personalized treatments. We call the envisioned approach Tracking Tomographic Microscopy (TTM). The stated problem emerges in the context of the optical or acoustical trapping and manipulation of microscopic specimens, as recently developed by the group of our designated Mercator fellow Prof. Dr. Monika Ritsch-Marte, Medical University of Innsbruck, since the generated force fields that are used for rotating the samples are specimen-dependent. The motion of the specimen is therefore characterized by a non-uniform rotation speed and a precession and nutation of the specimen's rotation axis that is periodic in nature, i.e. the same pose is achieved after a full rotation. We aim at solving this new pose estimation problem by suitable adaptations of structure-from-motion(SfM) algorithms. Our work program will develop the necessary components and modifications to feature detection, tracking, outlier rejection, data filtering and bundle adjustment algorithms such as to enable the reliable estimation of poses for tracking tomographic microscopy (TTM). Our strategy is the full exploitation of domain knowledge for solving this difficult task. Therefore, all steps of the SfM pipeline will be addressed under the new premises. Our goal is the development of a robust estimation pipeline that we aim to make available to the scientific community by means of extending an existing open source project.

• DFG Programme: Research Grants
• International Connection: Austria
• Cooperation Partner: Professorin Dr. Monika Ritsch-Marte