Phonetic Convergence in Human-Computer Interaction

• Applicant: Professor Dr. Bernd Möbius, since 1/2019
• Subject Area: General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2015 to 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 278805297

• Final Report Year: 2022

Final Report Abstract

The proposed research project aimed at the analysis, quantification, modeling, and evaluation of phonetic convergence in human-human and human-computer interactions. Phonetic convergence is defined as an increase in segmental and suprasegmental similarities between two speakers, presumably grounded in spontaneous phonetic adoption of speech characteristics of the interlocutor. Building on research on convergence in human-humancommunication, this project develops a quantitative model of phonetic convergence in spoken human-computer interaction and its application in a simulated spoken dialog system environment and, specifically, its speech synthesis component. Implications for the design of conversational interfaces in speech technology are inferred. We are now in a position to demonstrate that human experimental subjects show patterns of phonetic convergence when being exposed to synthetic voices. These patterns are qualitatively and quantitatively similar to the convergence patterns observed in human-to-human interaction. We have also implementated and evaluated an adaptive spoken language dialog system that enables convergence patterns observed in human-human interactions. A better understanding of the accommodation phenomena related to various acoustic-prosodic, temporal, and spectral features may further improve the performance of current spoken dialog system technology, leading to smoother conversational dialogs.

Publications

• A Computational Model for Phonetically Responsive Spoken Dialogue Systems. Interspeech 2017, 884-888. ISCA.
  Raveh, Eran; Steiner, Ingmar & Möbius, Bernd
  
• Shadowing Synthesized Speech — Segmental Analysis of Phonetic Convergence. Interspeech 2017, 3797-3801. ISCA.
  Gessinger, Iona; Raveh, Eran; Le Maguer, Sébastien; Möbius, Bernd & Steiner, Ingmar
  
• Convergence of Pitch Accents in a Shadowing Task. Speech Prosody 2018, 225-229. ISCA.
  Gessinger, Iona; Schweitzer, Antje; Andreeva, Bistra; Raveh, Eran; Möbius, Bernd & Steiner, Ingmar
  
• Studying Mutual Phonetic Influence with a Web-Based Spoken Dialogue System. Lecture Notes in Computer Science, 552-562. Springer International Publishing.
  Raveh, Eran; Steiner, Ingmar; Gessinger, Iona & Möbius, Bernd
  
• A Wizard-of-Oz experiment to study phonetic accommodation in human-computer interaction. In: 19th International Congress of Phonetic Sciences, S. 1475–1479
  I. Gessinger, B. Möbius, N. Fakhar, E. Raveh & I. Steiner
  
• Comparing phonetic changes in computer-directed and human-directed speech. In: Elektronische Sprachsignalverarbeitung 2019, Tagungsband der 30. Konferenz (Dresden), S. 42–49
  E. Raveh, I. Steiner, I. Siegert, I. Gessinger & B. Möbius
  
• Three’s a Crowd? Effects of a Second Human on Vocal Accommodation with a Voice Assistant. Interspeech 2019, 4005-4009. ISCA.
  Raveh, Eran; Siegert, Ingo; Steiner, Ingmar; Gessinger, Iona & Möbius, Bernd
  
• Differences in Gradient Emotion Perception: Human vs. Alexa Voices. Interspeech 2020, 1818-1822. ISCA.
  Cohn, Michelle; Raveh, Eran; Predeck, Kristin; Gessinger, Iona; Möbius, Bernd & Zellou, Georgia
  
• Phonetic Accommodation of L2 German Speakers to the Virtual Language Learning Tutor Mirabella. Interspeech 2020, 4118-4122. ISCA.
  Gessinger, Iona; Möbius, Bernd; Andreeva, Bistra; Raveh, Eran & Steiner, Ingmar
  
• Phonetic accommodation in interaction with a virtual language learning tutor: A Wizard-of-Oz study. Journal of Phonetics, 86, 101029.
  Gessinger, Iona; Möbius, Bernd; Le Maguer, Sébastien; Raveh, Eran & Steiner, Ingmar
  
• Phonetic accommodation to natural and synthetic voices: Behavior of groups and individuals in speech shadowing. Speech Communication, 127, 43-63.
  Gessinger, Iona; Raveh, Eran; Steiner, Ingmar & Möbius, Bernd
  
• Cross-Cultural Comparison of Gradient Emotion Perception: Human vs. Alexa TTS Voices. Interspeech 2022, 4970-4974. ISCA.
  Gessinger, Iona; Cohn, Michelle; Zellou, Georgia & Möbius, Bernd