A Generalised Approach to Learning Models of Human Behaviour for Activity Recognition from Textual Instructions

• Applicant: Professorin Dr.-Ing. Kristina Yordanova
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2016 to 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 314457946

Computational models for activity recognition aim at recognising the user actions and goals based on precondition-effect rules. One problem such approaches face, is how to obtain the model structure. To reduce the need of domain experts or sensor data during the model building, methods for learning models of human behaviour from textual data have been investigated. Existing approaches, however, make various simplifying assumptions during the learning process. This renders the model inapplicable for activity recognition problems. To address this problem, this project aims at developing a generalised methodology for learning the model structure from textual instructions. The methodology shall combine existing and novel methods for model learning.- A methodology for extracting the action semantics from text shall be developed. The methodology shall address the challenge of identifying causal relations between elements in texts with short and simple sentence structure.- To ensure the model generalisation and the incorporation of context information, methods for ontology learning shall be investigated. A core challenge here is the ontology extension based on causal, spatial, and functional properties of the entities in the problem domain.- To learn the model semantics, methods for language grounding shall be investigated. The semantics shall be represented in terms of precondition-effect rules. The resulting methodology shall address the issues associated with learning these rules and translating them into an appropriate for activity recognition format. It shall also address the problem of learning an optimal model through reinforcement learning methods.- To evaluate the methodology, the learned models shall be applied to various activity recognition tasks and their performance compared to that of handcrafted models.If successful, the methodology will reduce the time and resources needed for developing computational models of human behaviour for activity recognition.

• DFG Programme: Research Grants