Semantics-driven Eye Gaze Prediction in Online Collaboration (SemCoGaze)

• Applicant: Professor Dr. Steffen Staab
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Methods in Artificial Intelligence and Machine Learning
• Term: since 2026
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 578894011

By understanding and predicting eye movements, we can better support users of graphical user interfaces (GUIs) with diverse cognitive and motor abilities. Eye movements are influenced by contextual factors—such as the tasks and social contexts of users and the meaning of the perceived content. Existing prediction models take little account of these contextual factors, which are ubiquitous in GUIs.Our research goal is to use context to improve algorithms for predicting eye movements. To this end, we define three sub-goals that integrate context information of increasing complexity:(Z1) Integrating semantics of tasks and graphical user interfaces;(Z2) Integration of a pragmatic model for parasocial interactions;(Z3) Integration of a pragmatic model for collaborative interactions.In three corresponding work packages (WP1-WP3), we collect experimental data, develop new algorithms, and validate and compare new algorithms with existing ones.WP1 investigates deep learning models that analyze natural language descriptions of tasks and the layout and content of GUIs and integrates them into novel eye movement prediction algorithms. Neural network encoders map different input data streams into common latent spaces, and neural network decoders (transformer and diffusion models) autoregressively predict eye and mouse movements.WP2 develops new algorithms for the context of parasocial interactions based on the results of WP1. Parasocial interactions are characterized by a strong asymmetry between active users, who dominate communication, and passive users. To this end, we want to model how the attention of passive users is coupled with that of active users and how this coupling correlates with communication success.WP3 extends these results to the contexts of collaboration and cooperation, where all users are active. Cooperating users share the workload, work independently, and finally merge their results. Collaborating users work simultaneously on the same object. We intend to predict all users' gaze and mouse movements using observations of each other's previous gaze and mouse movements. In particular, we will investigate how gaze patterns can distinguish collaboration and cooperation, and how the transitions between these two forms of social interaction can be recognized.By analyzing eye movements in visual and task-oriented contexts and multi-user scenarios, SemCoGaze will make a fundamental contribution to our understanding of attention, which will help support users in their interactions with computers in the future. Data and software resulting from the project will be published in compliance with data protection regulations.

• DFG Programme: Research Grants