Audio-visual prosody of whispered and semi-whispered speech
Final Report Abstract
The studies conducted within this project contribute a novel perspective to the current body of research. The results indicate that communicatively marked situations elicit distinct differences in both the oro-facial expressions and the acoustic signals produced by speakers. Specifically, both German and Farsi speakers exhibited more pronounced facial expressions, such as raised eyebrows, larger lip apertures, and wider eye openings when whispering. Similarly, more intense oro-facial expressions were observed when speakers were unable to see each other, either completely (for German speakers) or partially, due to facial masks (for Farsi speakers). Furthermore, we identified cumulative effects: the most pronounced oro-facial expressions occurred when speakers were whispering and unable to see each other. Most findings suggest compensatory mechanisms, such as more pronounced gestures and longer durations compensating for the absence of the fundamental frequency, supporting the trade-off hypothesis. However, since gestures were also enhanced when the listener was invisible, we concluded that these gestures are not merely for the listener’s benefit (supporting the hand-in-hand hypothesis), but rather serve an overarching communicative goal. The major surprise was the pandemic, which forced us to seek different methods and solutions from those proposed in the project. Instead of German speakers, we first recorded Farsi speakers, which was possible at that time in Iran, and gained an additional group of informants for our experiment. At that time, it was impossible to make recordings in Berlin. Later, we also conducted experiments with German speakers as indicated in the project proposal. This allowed us to compare the orofacial expressions of Farsi and German speakers. We also transitioned from the motion capture method (with markers glued to the participants’ faces) to a more efficient method, i.e., Open Face, which made it possible to measure more oro-facial movements recordings in a more ecologically valid way and by using more measurement points. Żygis, Marzena : Psst… Was passiert, wenn wir flüstern? Lange Nacht der Wissenschaften. Leibniz Gemeinschaft, Berlin, 15.06.2019
Publications
-
How do our eyebrows respond to masks and whispering? The case of Persians. Interspeech 2022. ISCA.
Sardhaei, Nasim Mahdinazhad; Zygis, Marzena & Sharifzadeh, Hamid
-
Communicative constraints affect oro-facial gestures and acoustics: Whispered vs normal speech. The Journal of the Acoustical Society of America, 153(1), 613-626.
Żygis, Marzena & Fuchs, Susanne
-
Orofacial expressions and acoustic cues in German questions. Proceedings of the 20th International Congress of Phonetic Sciences, 4140-4144. Paper ID: 441.
Żygis, M., Sardhaei, N. & M. Krifka
-
Facial expressions in different communication settings: A case of whispering and speaking with a face mask in Farsi. Language and Cognition, 16(4), 1639-1673.
Mahdinazhad, Sardhaei Nasim; Żygis, Marzena & Sharifzadeh, Hamid
