Video-based automated pain detection exploiting compositional and temporal characteristics of action units (PainFaceReader)

• Applicants: Dr.-Ing. Jens-Uwe Garbas; Professor Dr. Stefan Lautenbacher; Professorin Dr. Ute Schmid
• Subject Area: Biomedical Systems Technology ; General, Cognitive and Mathematical Psychology ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2018 to 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 405630557

Goal of the project is to develop and evaluate a system for automatic pain recognition for everyday clinical use. Accordingly, the classification must be both robust and accurate at the individual level as well as comprehensible for medical staff. In this follow-up application, the work delayed by Corona will be continued and expanded. Based on already achieved results, further research questions to achieve the overall goal will be addressed. Our approach is based on the description of facial expressions by action units (AUs) which are descriptors of movements of facial muscles as defined in the Facial Action Coding System (FACS). Specific constellations of AUs and their intensities are then used as indicators for the presence of certain mental states. Currently, FACS coding is mainly done manually, by trained FACS coders, which is very cost-intensive. The aim of the project is to develop a camera-based online monitoring and analysis system that uses video recordings of the face (1) to continuously detect the occurrence of AUs and their intensities, and (2) based on this, to recognise the presence of pain as distinct from other aversive emotions such as anger and disgust.In an interdisciplinary group of electrical engineering, computer science and psychologists experienced in pain research, the PainFaceReader is to be developed as a novel approach that meets the requirements described.In the first three years, the planned psychological data collection could only be carried out to a very limited extent due to Corona. Accordingly, the planned experiments and studies are to be completed in the follow-up application. Up to now, the Fraunhofer Insitute used public data bases to train an approach for determining and monitoring. Likewise, the Bamberg computer scientists did use synthetic data to train logic-based classifiers. The approaches for AU detection and pain classification are now to be evaluated on the newly collected data. A systematic comparison of existing approaches for AU identification is planned with regard to the recognition accuracy for the pain-relevant AUs and their combinations. The AU identification approach developed in the previous application will be compared with commercial approaches. In particular, analyses will be carried out on the robustness of the AU detection for groups of people with different characteristics such as age and gender. The own approach will be further developed by exploiting correlations between AUs across different emotions. Finally, uncertainties of AU detection in the classification of pain will be taken into account and an explanatory component for pain detection in the clinical context will be added to the system.

• DFG Programme: Research Grants