Planning and executing robotic actions using simulated image sequences created by generative deep neural networks

• Applicant: Professor Dr. Florentin Wörgötter
• Subject Area: Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2019 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 417069796

• Final Report Year: 2023

Final Report Abstract

The central idea behind this project was to understand to what degree it is possible to arrive at nontraditional planning methods by employing neural networks instead of commonly used symbolic planners and to try to (better) understand the underlying processes, by analyzing the algorithms and the underlying networks. 1. The core of this project has been investigating whether it is possible to perform formal planning (e.g. for robotics applications by planning an action sequence) by using “mental imagery”, where, instead of symbolic plans, a sequence of images is created to represent a pictorial plan. This starts from the real scene and – by a generative process – the system is then creating a new image that shows the outcome of an action, without executing it; and so on until plan completion (or failure). Here we also showed how to “understand” the intrinsic processes of this system in “human terms”, by being able to translate a pictorial (mental) plan back into a human-readable symbolic plan. 2. In addition, we have investigated the question of how to perform planning, here of an optimal path, together with (neuronal) learning by ways of a novel artificial neural network. This is a novel algorithm with the same computational properties and the same algorithmic complexity as the classical Bellman-Ford algorithm, but now we can add learning to it “on the fly”, which has not been achieved so far. Here we also provided analytical insights into the workings of this new algorithm. 3. Lateral to these two core questions was another investigation, where we addressed the question of neural network performance and how to gain better understanding of how complex neural networks operate. This part of the investigation had emerged in the course of the project development and had – initially – not been planned in the grant proposal. It links to the above stated question that part of the here-pursued research was targeted at the problem to better understand the functions of the algorithms. Here we addressed this by investigating how networks and their layers are driven to their decisions using several quantitative measures.

Publications

• Interpreting the decisions of CNNs via influence functions. Frontiers in Computational Neuroscience, 17.
  Aamir, Aisha; Tamosiunaite, Minija & Wörgötter, Florentin
  
• Simulated mental imagery for robotic task planning. Frontiers in Neurorobotics, 17.
  Li, Shijia; Kulvicius, Tomas; Tamosiunaite, Minija & Wörgötter, Florentin
  
• Combining Optimal Path Search With Task-Dependent Learning in a Neural Network. IEEE Transactions on Neural Networks and Learning Systems, 36(1), 498-509.
  Kulvicius, Tomas; Tamosiunaite, Minija & Wörgötter, Florentin