user adaPtive Artificial iNtelligence fOR humAn coMputer interAction
Final Report Abstract
The key concept of this project is “user adaptive AI in the context of human-computer interaction”. When people talk to other people, they change their verbal and nonverbal communication behaviors according to those of the partner. Therefore, to improve useradaptivity in human-agent interaction, the system needs to recognize human communication signals and generate agent’s behaviors according to the status of the communication. Towards this goal, this project addressed issues of collecting and analyzing corpus, developing conversational agents, and machine learning techniques. In corpus studies, we collected multi-cultural corpus in two scenarios: Motivational Interviewing (MI) and NoXi. MI is a collaborative goal-oriented communication aiming to strengthen personal motivation for a specific goal. PANORAMA partners collected multi-cultural MI corpus in French, English and Japanese, and annotated the corpora using standard MI coding schemes. The second scenario is NoXi, which collected expertnovice interaction in Germany, French, English, and Japanese. The European NoXi was prepared for an international challenge of human social behavior analysis tasks with gold-standard annotation. We also developed the NOVA tool to support manual annotation processes. We used NOVA as the platform of manual annotation in the PANORAMA project, and the MI and NoXi corpora can be accessed via the NOVA tool. As the component of conversational agents, we developed methods of recognition and generation of social signals as listed below: We investigated interruption mechanism using NoXi corpus and developed a generative model for facial expressions and head rotations, exhibited by the interrupter during the interruption phase. - We investigated effectiveness of 3D face models, and provided a comprehensive analysis of their current capabilities and limitations when compared to state-of-the-art models that rely on 2D face image sequences. - We participated in the Affective Vocal Bursts Competition in ACII 2022, and in the A-VB Type Task, which aimed to automatically distinguish between different types of vocal bursts, we achieved the first-place position. - Analyzing MI corpus, we proposed a method for detecting Change Talk, which is a statement by a client revealing consideration of, motivation for, or commitment to change. We also proposed a method that generates facial expression dynamics in MI using conditional diffusion model. To improve verbal communication with users, we proposed dialogue managers with user-adaptive dialogue strategies. In particular, we focused on exploiting knowledge graph and dialogue generation methods considering semantic representation in interviewing interaction. In addition to the studies focusing on specific scenarios and domains, we proposed machine learning techniques that can be widely applied to a variety of human behavior data. First, we created a multitask learning algorithm using different kinds of multimodal social signal corpora. We also proposed an efficient transfer learning algorithm for adapting to the user’s multimodal behaviors.
Publications
-
Adaptive Artificial Personalities. The Handbook on Socially Interactive Agents, 155-194. ACM.
Janowski, Kathrin; Ritschel, Hannes & André, Elisabeth
-
AI for a Sustainable Future. 3rd Japanese-German-French AI Symposium: AI for Planetary Challenges in the Anthropocene,' held in Tokyo from October 27 to 28, 2022
Elisabeth André
-
Bridging the Gap: End-to-End Domain Adaptation for Emotional Vocalization Classification using Adversarial Learning. Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, 95-100. ACM.
Schiller, Dominik; Mertes, Silvan; van Rijn, Pol & André, Elisabeth
-
MultiMediate '23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions. Proceedings of the 31st ACM International Conference on Multimedia, 9640-9645. ACM.
Müller, Philipp; Balazia, Michal; Baur, Tobias; Dietz, Michael; Heimerl, Alexander; Schiller, Dominik; Guermal, Mohammed; Thomas, Dominike; Brémond, François; Alexandersson, Jan; André, Elisabeth & Bulling, Andreas
-
Phoneme-Based Multi-task Assessment of Affective Vocal Bursts. Communications in Computer and Information Science, 209-222. Springer Nature Switzerland.
Hallmen, Tobias; Mertes, Silvan; Schiller, Dominik; Lingenfelser, Florian & André, Elisabeth
-
Shonan Meeting 202, 22 Oct - 26 Oct 2023, Building and Evaluation Adaptation Mechanisms for Socially Interactive Agents
Catherine Pelachaud
-
Shonan Meeting 202, 22 Oct - 26 Oct 2023, Shonan, Japan. Social and Motivational Personalised Interactions
Jean-Claude Martin
-
Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?. IEEE Transactions on Affective Computing, 15(2), 535-548.
Tellamekala, Mani Kumar; Sümer, Ömer; Schuller, Björn W.; André, Elisabeth; Giesbrecht, Timo & Valstar, Michel
