Multimodal presentation and exploration of time-dependent real-time data by blind people

• Applicant: Professor Dr. Gerhard Weber
• Subject Area: Human Factors, Ergonomics, Human-Machine Systems ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 514258978

In order to monitor complex data that is generated in real time, it is usually visualized. To integrate blind people into the workspace of data monitoring, an adequate sonification and haptification is missing. In this context, a combination of Braille as well as speech synthesis is already used by blind employees to interact with graphical user interfaces, so that collaboration with sighted people is also possible on the basis of verbalization of graphical content (menus, text fields, etc.). The main goal of the project is to design a more complex multimodal, non-visual software concept for capturing and exploring real-time data streams with currently available mobile devices and wearables to enable blind people to analyze the data independently. Previous work has considered multimodality through sonfication and haptification, each often in isolation. This project aims to expand non-visual access to information visualization by mergingverbal and non-verbal representations. This requires characterizing the complexity of features as well as incorporating temporal requirements from the overlay of real-time data with non-visual interaction via keyboard with screen readers. Passive perception through, for example, vibration via wearables or sonified data features will be integrated with active actions for exploration and analysis and incorporated into the rest of the user's activities. User-centered design is a proven method to develop software applications that are tailored to the needs of the target audience. We want to follow this design method in the proposed project in order to adapt the choice of modalities to the needs of the users, depending on the use of screen reader modalities. For this, it is necessary to involve blind and sighted people from the very beginning. This requires a combination of conceptual work and empirical research. The work plan consists of a total of four work packages pursuing three different use cases and an integrated novel concept. A total of eight sub-studies and one workshop are planned within three years.

• DFG Programme: Research Grants