Learning to Interact

• Applicant: Professor Dr. Andreas Bulling
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2026
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 569311510

The ability for an artificial intelligent (AI) system to effectively interact with its users requires in-depth understanding of their interactive behaviour, i.e. the specific sequence of input actions that users perform when they interact with a user interface (UI). In human-computer interaction (HCI), dominant approaches to modelling interactive behaviour include descriptive models (such as keystroke-level models, KLM) or predictive models (such as Fitts' Law) as well as methods based on machine learning. However, all of these models are either limited to task execution time as the only measure of user performance, assume expert users who do not make errors, only consider routine tasks, or they require highly-controlled setups and are application- or task-specific, thus severely limiting their generalisability. The goal of this project is to pioneer a radically different approach to interactive user behaviour modelling. Inspired by research in natural language processing, we aim to learn semantically rich and reusable interaction embeddings in an unsupervised manner from large-scale unlabelled data. Similar to natural language, interactive behaviour is temporal/sequential in nature and has rich internal structure and inter-dependencies between its basic building blocks. In addition, interactive behaviour exhibits a large amount of consistent global characteristics across users. Despite these striking similarities, however, embeddings of interactive behaviour remain under-explored in HCI. To achieve this goal, this project will address four complementary research challenges at the intersection of AI and HCI: 1) Creating a large-scale and variable interaction corpus suitable for training interaction embeddings. 2) Developing computational methods to encode behavioural data, analyse its similarity with natural language in terms of its structure and internal inter-dependencies, and identify basic and higher-level building blocks of the "language of interaction". 3) Developing unsupervised methods to learn and analyse interaction embeddings on their own and jointly with embeddings that encode UI semantics. 4) Applying these embeddings for challenging and novel downstream tasks, such as extracting key actions from interactive behaviour, question answering for interactive behaviour, and automatic UI design optimisation.

• DFG Programme: Research Grants