Semantic and Local Computer Vision based on Color/Depth Cameras in Robotics (SeLaVi)

• Applicant: Professor Dr. Dominik Henrich
• Subject Area: Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics
• Term: from 2019 to 2023
• Website: Homepage
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 405548722

For modern robots, recognizing objects in their environment is a key skill that enables useful and flexible actions. For this purpose, usually one or more camera images of the scene are evaluated and an internal representation of the environment for the robot is built. However, image-based object recognition is confronted with some difficulties: on the one hand, geometrically simple and slightly textured objects, as they occur in many applications, are barely recognized. On the other hand, (partially) hidden objects in the scene are perceived worse or not at all. In addition, the sensory information usually describes only the geometry and location of the individual objects but not their semantic function or relationships with each other (e.g., "A is on B"). By contrast, perception as a long-term goal should "understand" the environment so that the various objects of a complex scene can be meaningfully manipulated by means of robots. In addition, few local views on the scene should be sufficient to create the most complete global environmental representation possible.The proposed research project "SeLaVi" develops and examines new concepts for image-based understanding of a scene. As a new and unique basic approach serve geometric models, which represent the objects by few surface patches (Boundary Representations, BReps) and which are generated from one or more depth images. This ensures a significantly higher storage and computational efficiency of the method than is possible with the common point clouds or triangular net-works. Based on the BRep and on additional color information from the scene, the objects of an object database are recognized. The recognition of the static objects should work with few local views on the scene and be as robust as possible against other moving objects (for example humans). The world model created in this way is then extended by semantic relations between the objects in order to enable manipulation by a robot arm. In addition, the semi-automatic generation of the object database by the user is considered. The potential fields of application range from autonomous ser-vice robots, programming-to-programming and human/robot cooperation, to industrial automation (e.g. handle-in-the-box).

• DFG Programme: Research Grants