WeGaze: Large-Space Gaze Analysis via Multi-camera System for Social Interactions

• Applicant: Dr. Senya Polikovsky, Ph.D.
• Subject Area: Methods in Artificial Intelligence and Machine Learning ; General, Cognitive and Mathematical Psychology ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Social Psychology, Industrial and Organisational Psychology
• Term: since 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 563112506

Ocular metrics are essential for understanding human cognition, behavior, and interpersonal dynamics. Tracking multi-person ocular dynamics in large spaces in a non-invasive manner is a challenging technical problem. Current methodologies often rely on intrusive wearable trackers that can disrupt natural behavior and authentic interactions among individuals. While multi-camera setups hold significant promise for improving accuracy and robustness in complex settings, gaze estimation with multi-camera systems has yet to be thoroughly investigated in research. The ultimate objective of WeGaze is to expand the possibilities of gaze and ocular dynamic research by developing a novel system for detecting and analyzing ocular dynamics of multiple humans with multi-cameras in a large space for naturalistic settings. Our steps involve: (1) Leveraging our unique expertise in the multi-view RGB camera system and incorporating the professional eye tracker, we will collect an advanced dataset from multi-view cameras with ground-truth. (2) With the collected data, we will develop a new gaze estimation algorithm for multi-view camera settings. (3) We design a new setup to capture interactions involving multiple participants in large-spaces. (4) We extend the developed algorithm to accommodate multi-person scenarios, enabling high-level eye movement analysis, including shared and mutual gaze behavior, in dynamic, room-scale environments. (5) Finally, we will evaluate the developed model in an environment, such as a room equipped with multi-cameras where multiple people are having natural conversations. The developed algorithm will be open source and distributed initially to the UGaze network. To ensure its usability, we will establish accuracy criteria focusing on gaze direction, eye blinks and interaction-based labels such as mutual gaze and gaze-sharing. This project will help expand the current limits of eye tracking technology and enable application of gaze studies in multi-person, naturalistic settings.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2481: Understanding Gaze
• International Connection: Netherlands
• Cooperation Partner: Professor Xucong Zhang, Ph.D.