The Interactive Cookbook

• Applicant: Professor Dr. Alexander Koller
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 461220770

The goal of this project is to develop an "Interactive Cookbook": a spoken dialogue system that guides users through the process of cooking a dish. The system will analyze individual recipe texts for a given dish offline and aggregate them into a symbolic recipe graph, which captures alternative ways in which the recipe can be explained and carried out. At dialogue time, a natural language understanding (NLU) component will process user requests regarding the dish, and a natural language generation (NLG) component will generate step-by-step instructions in English or German based on the recipe graph. The dialogue system will adapt the level of detail at which it presents the recipe to the user's needs.In developing such an Interactive Cookbook, we will face a number of research challenges. First, cooking recipes are a rather specific type of text, and NLU methods trained on main- stream corpora will have to be adapted to the recipe domain. We will also consolidate different recipes for the same dish into a single representation, so we can switch to different levels of detail as required for the user.Second, we will have to refine neural word embeddings to capture domain-specific meaning distinctions: “salt” and “sugar” are distributionally very similar, but confusing one for the other in a recipe will drastically change the dish.Finally, we will have to move beyond current methods in NLG: grammar-based methods run into coverage problems when explaining arbitrary recipes, and neural methods such as those in recent approaches to abstractive summarization struggle to produce language which conveys a given meaning. One focus of the project will therefore be to generate instructions which are semantically true to the underlying recipes, by conditioning a neural NLG system on symbolic recipe graphs.

• DFG Programme: Research Grants
• Co-Investigator: Dr. Lucia Donatelli, Ph.D.