Analysing Legislative Politics with Aligned Text, Audio, and Image Data

• Applicant: Professor Dr. Christian Stecker
• Subject Area: Political Science ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 545741586

Political science has extensively studied the communication of political actors, e.g., in press releases, electoral manifestos, or parliamentary speeches. But political communication is more than the written word alone. Non-verbal communication greatly matters in politics, as exemplified in speeches or campaign ads. Our project explicitly considers this non-verbal information and promises novel insights by linking multimodal data to meaningful political science concepts such as salience or inter-party mood. Recent technological advancements allow us to tap into this potential fruitfully. We exploit the epistemic potential of multimodal political communication, i.e., studying text, audio, and video in a combined manner. Our project joins novel approaches to the study of politics and creates an original multimodal dataset based on video recordings of plenary speeches, develops innovative metrics of essential concepts, and leverages these to advance research in different areas. We make three main contributions: For our DATA CONTRIBUTION, we curate all parliamentary debates held in the 16 German state parliaments and the Bundestag between 2005 and 2023. We extract discrete features from this data, for example, spoken words, vocal pitch, facial expressions, and body language. Aligning all this data, we create a fully searchable multimodal dataset of more than 25,000 hours of speeches. In a MEASUREMENT CONTRIBUTION, the project automatically annotates the eight key political emotions: anger, fear, disgust, sadness, joy, pride, enthusiasm, and hope. We also quantify arousal and valence. Building on this, we measure two prominent political science concepts, i.e., issue salience and inter-party mood. Deeply rooted in a multimodal understanding of political communication, we offer a more nuanced and precise approach than the current state-of-the-art. Our RESEARCH CONTRIBUTION utilizes the novel dataset and measures to advance three research areas: the influence of populist parties on parliamentary debates, the dynamics of issue competition, and the nature of inter-party mood in a multi-level political system. These studies employ multimodal salience measures to understand non-verbal communication of issues, interpret inter-party mood through the speaker’s gaze, and assess the effects of populist parties using multimodal emotion measurements. Substantively, the research contribution benefits from the methodological advantages of Germany's multi-level system, where 16 state parliaments offer various political configurations while erroneous variance is minimized. In addition, the Bundestag allows for analyzing multi-level effects. In sum, our project advances the study of politics by applying a multimodal framework that captures a richer, more nuanced picture of parliamentary communication.

• DFG Programme: Research Grants
• International Connection: United Kingdom
• Cooperation Partner: Dr. Christian Arnold