HFVSA: Human Focused Visual Scene Understanding

• Applicant: Professor Dr. Jürgen Gall
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2013 to 2017
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 229087185

A meaningful interpretation of an observed scene covers many important aspects like the similarity of observed objects, the categorization of objects, the functionality of objects, and the action of humans. While visual data like images or videos can be efficiently interpreted by humans, computers still struggle to solve these tasks. This indicates that existing learning procedures and representations of categories are insufficient for scene understanding. For instance, a detector for a specific object class is typically trained on thousands of images containing class examples. Although such a detector might see even more examples than a human, it is not able to learn the intrinsic similarity of the object class and thus lacks the ability of humans to categorize objects reliably. Indeed, humans do not learn only from images but observe a continuous data stream which includes additional cues like hand motion and transformations of the object during object manipulation. In this project, we aim to investigate the facilitative effect of observing human-object interactions for scene understanding. The main focus will be on understanding and learning the intrinsic similarity of visual categories by taking these additional cues into account. Finally, we plan to develop a system that processes automatically visual data streams, extracts the human motion, infers the objects humans interact with, and learns the similarity of objects and their functionality.

• DFG Programme: Independent Junior Research Groups