TherapAI - Analysis of nonverbal emotional expression in psychotherapy through artificial intelligence

• Applicants: Professorin Dr. Elisabeth André; Professor Dr. Wolfgang Lutz
• Subject Area: Personality Psychology, Clinical and Medical Psychology, Methodology ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 493169211

Background: Focusing on emotions in psychotherapy is highly relevant for therapeutic outcome and process, especially in emotional disorders like depression. Beyond patient’s or therapist’s individual emotions, their emotional interaction (e.g., emotional synchrony; ES) plays an important role. ES can provide information relevant to the therapeutic process, especially for patients with disturbed perception and expression of emotions, as is the case with depression. However, studies of emotions and ES have mainly been based on self-reports or small samples, because manual ratings are time-consuming. Artificial intelligence (AI) video analysis software such as the Nonverbal Behavior Analyzer (NOVA) can automatize, facilitate, and improve the rating process, providing new opportunities to examine emotions and emotional interactions in psychotherapy.Aim: The aim of this project is to examine how patients’ and therapists’ emotions and ES, automatically assessed by NOVA, are related to outcome, dropout, session-to-session change, and process variables (coping skills, therapeutic relationship, emotional involvement) in a sample of patients with depression (F32, F33, diagnosed with the SCID-5). Simultaneously, NOVA will be adapted to the needs of clinical application based on the findings gained from practical application over the course of the study. In particular, user-friendliness and comprehensibility of the functions will be improved. To facilitate replicability of the study results by other researchers, the applied models will be made freely available.Method: The total sample consists of N = 200 therapists and patients with approx. 1,800 therapy videos (á 50 minutes). NOVA will be trained to detect patients’ and therapists’ emotions and ES in the videos. To this end, trained human raters will manually code valence and arousal of the emotional expression in four sessions per dyad. Based on these manual ratings, NOVA will learn to automatically rate emotions in videos. Models will be trained in 70% (n = 140) of the data and tested in the remaining 30% (n = 60) ten times to select the best model. Then, NOVA will use the final model to automatically rate patients’ and therapists’ emotions in all available videos of the first ten sessions of the 200 dyads. Afterwards, ES will be calculated. The approx. 1,800 sessions (10% missings expected) will be used to test how the average levels of emotions and ES are related to outcome, symptom reduction, dropout, session-to-session change, and process variables (coping skills, therapeutic relationship, emotional involvement) using multilevel linear models and logistic regressions.Discussion: Due to the importance of emotions for psychological disorders and therapies, an automated analysis of emotions developed in this project expands the possibilities of psychotherapy research regarding the measurement and prediction of treatment processes and outcome.

• DFG Programme: Research Grants