PIPE: Probabilistic Models of Instructions, Perception and Experience - Representation, Learning and Reasoning

• Applicant: Professor Michael Beetz, Ph.D.
• Subject Area: Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2017 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 322037152

• Final Report Year: 2023

Final Report Abstract

In P IPE, we designed, realized and investigated joint Bayesian models for the compact representation and efficient learning of, as well as reasoning about (I) task interpretation, (P) perception and (E) execution for mobile robots. We developed a framework for joint probabilistic learning and reasoning, called joint probability trees (JPT), which makes learning of and reasoning about joint probability distributions tractable for practical applications. We developed probabilistic hybrid action models, which represent joint distributions over a robot’s belief state and actions, their parameters and perceptual events, which allow reasoning about how actions can be performed in order to bring about the desired effects, or to anticipate undesired ones. We have successfully shown the application and strengths of the PIPE approach in milestone meetings of the collaborative research center EASE. Here, we successfully applied the PIPE framework for learning from Instructions, Perception and Experience to learn motion parameters of a fetch-and-place action, and to learn specialized plan variants so the robot was able to execute those actions faster and more reliably. The PIPE learning framework has been released as an open-source project, which is actively used and developed by the community.

Publications

• Self-Specialization of General Robot Plans Based on Experience. IEEE Robotics and Automation Letters, 4(4), 3766-3773.
  Koralewski, Sebastian; Kazhoyan, Gayane & Beetz, Michael
  
• Learning Motion Parameterizations of Mobile Pick and Place Actions from Observing Humans in Virtual Environments. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 9736-9743. IEEE.
  Kazhoyan, Gayane; Hawkin, Alina; Koralewski, Sebastian; Haidu, Andrei & Beetz, Michael
  
• Joint Probability Trees
  Nyga, Daniel