(DEEP) Deep Emotion Processing for Social Agents Combining Social Signal Interpretation
and Computationally Modeling User Emotions

• Applicants: Professorin Dr. Elisabeth André; Dr.-Ing. Patrick Gebhard
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2018 to 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 392401413

• Final Report Year: 2023

Final Report Abstract

The primary aim of the DEEP project was to create a novel comprehensive computational model intertwining social cues, contextual elements, and internal emotional processes. This model was envisioned to a) lay a foundation for more sophisticated computational representations of human emotions for related future research and b) empower research about next-generation socially interactive agents by heightening their awareness and sensitivity to users’ emotional states, fostering adaptability to users’ varying affective contexts. Overall, the project successfully met its objectives and all milestones. A specific use-case adaptation allowed the project partners to delve further into pertinent topics central to the DEEP project’s core. Notably, the project yielded 25 publications. From these, nine are published in the most important conference in the field of affective computing, namely the International Conference on Affective Computing and Intelligent Interaction (ACII), and two are published in the most important journal namely the IEEE Transactions on Affective Computing. Results were also shared with general public whenever feasible, like in the Manager Magazin or by numerous invited talks by the PIs on related events both nationally and internationally, e.g., most recently the symposium "Roboter als Empathisches Gegenüber" in Loccum, Germany or the expert meeting about "Conversational Qualities in Dyadic and Group Interactions" in Shonan, Japan. The acquisition of an industry project underscores the broad applicability of the DEEP project’s outcomes across diverse domains. This partnership demonstrates the high relevance of the project’s results in various application areas beyond the initial scope.

Publications

• Can Social Agents elicit Shame as Humans do?. 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), 164-170. IEEE.
  Schneeberger, Tanja; Scholtes, Mirella; Hilpert, Bernhard; Langer, Markus & Gebhard, Patrick
  
• PARLEY. Companion Proceedings of the 24th International Conference on Intelligent User Interfaces, 35-36. ACM.
  Schneeberger, Tanja; Gebhard, Patrick; Baur, Tobias & André, Elisabeth
  
• eXplainable Cooperative Machine Learning with NOVA. KI -Künstliche Intelligenz, 34(2), 143-164.
  Baur, Tobias; Heimerl, Alexander; Lingenfelser, Florian; Wagner, Johannes; Valstar, Michel F.; Schuller, Björn & André, Elisabeth
  
• “Let me explain!”: exploring the potential of virtual agents in explainable AI interaction design. Journal on Multimodal User Interfaces, 15(2), 87-98.
  Weitz, Katharina; Schiller, Dominik; Schlagowski, Ruben; Huber, Tobias & André, Elisabeth
  
• Stress Management Training using Biofeedback guided by Social Agents. 26th International Conference on Intelligent User Interfaces, 564-574. ACM.
  Schneeberger, Tanja; Sauerwein, Naomi; Anglet, Manuel S. & Gebhard, Patrick
  
• Do Deep Neural Networks Forget Facial Action Units?—Exploring the Effects of Transfer Learning in Health Related Facial Expression Recognition. Studies in Computational Intelligence, 217-233. Springer International Publishing.
  Prajod, Pooja; Schiller, Dominik; Huber, Tobias & André, Elisabeth
  
• GANterfactual—Counterfactual Explanations for Medical Non-experts Using Generative Adversarial Learning. Frontiers in Artificial Intelligence, 5.
  Mertes, Silvan; Huber, Tobias; Weitz, Katharina; Heimerl, Alexander & André, Elisabeth
  
• Unraveling ML Models of Emotion With NOVA: Multi-Level Explainable AI for Non-Experts. IEEE Transactions on Affective Computing, 13(3), 1155-1167.
  Heimerl, Alexander; Weitz, Katharina; Baur, Tobias & Andre, Elisabeth
  
• Using Explainable AI to Identify Differences Between Clinical and Experimental Pain Detection Models Based on Facial Expressions. Lecture Notes in Computer Science, 311-322. Springer International Publishing.
  Prajod, Pooja; Huber, Tobias & André, Elisabeth
  
• The Deep Method: Towards Computational Modeling of the Social Emotion Shame Driven by Theory, Introspection, and Social Signals. IEEE Transactions on Affective Computing, 15(2), 417-432.
  Schneeberger, Tanja; Hladký, Mirella; Thurner, Ann-Kristin; Volkert, Jana; Heimerl, Alexander; Baur, Tobias; André, Elisabeth & Gebhard, Patrick