Verbal instructing of sensor-based robots (VerbBot)

• Applicant: Professor Dr. Dominik Henrich
• Subject Area: Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics
• Term: from 2017 to 2023
• Website: Homepage
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 320825892

The programming of robots is a key step in their start-up and further use in a variety of applications (e.g., industrial, laboratory, services). A robot program consists mainly of a sequence of instruc-tions (e.g., robot movements, tool commands) with the associated parameters (e.g., target position, force values) and a control structure for controlling the program sequence (e.g., repetitions, branches). The creation of such programs is time consuming and requires expertise. Even worse is the programming, if additionally sensor data must be evaluated to perceive the current environmen-tal situation (e.g., workpiece position, contact forces) to fulfill the task. Even advanced program-ming concepts, such as Graphical Programming or Programming By Demonstration, are only of limited help here. Programming of sensor-based robot systems thus represents a bottleneck for their efficient and flexible use in present and in future applications represent (e.g., household, work-shop, entertainment).The vision of this project is the intuitive programming of sensor-based robots through natural lan-guage. As a first step, here the usual robot commands (e.g., moveStraight, openGripper) are re-placed by verbs from the human language (e.g., TO LIFT or TO CARRY). For this purpose, first the semantic meaning of the verbs must be understood in more detail, to ensure that technical sup-port for the robot. (What is, e.g., the difference between TO PUSH and TO SHOVE?) Also a verb, to be used as a command for the robot, must be provided with parameters. (Whereto, e.g., do you want a workpiece to be pushed?) It must also be extracted additional, often only implicitly available context information from the situation. (What is, e.g., the weight of the object to be shoved?) In case of doubts, queries to the user or proactive actions may help. Finally, the instruction is to be executed sensor-based and any error situations need to be corrected (possibly in dialogue).

• DFG Programme: Research Grants