Safe Reinforcement Learning under Uncertainty for Hybrid Separation Processes with Recycles in Chemical Engineering

• Applicants: Professor Dr.-Ing. Sergio Lucia Gil; Professor Dr.-Ing. Jens-Uwe Repke
• Subject Area: Chemical and Thermal Process Engineering ; Automation, Mechatronics, Control Systems, Intelligent Technical Systems, Robotics
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 466380688

Online methods for the optimal operation of chemical processes, such as nonlinear model predictive control (NMPC), are desirable to save resources and costs. They are typically based on rigorous, first-principle, and dynamic models. Unfortunately, solving the resulting optimization problems in real-time is often not possible if the models include complex behavior such as start up of plants, preventing the application of NMPC to real chemical processes. The increasing availability of large amounts of data opens the door to operational strategies based on it, such as reinforcement learning (RL), but some critical challenges of RL, such as the rigorous consideration of process constraints or the large amount of data required, prevent RL from being applied to chemical processes. In the first phase of this project, we addressed these challenges by replacing the rigorous model by a surrogate which is used in an NMPC controller. An RL algorithm that interacts with a rigorous model adapts the NMPC problem to achieve optimal performance. We demonstrated this approach for the batch distillation of ethanol and water. At TU Berlin (applicant 1), the batch column was built and operated, and a dynamic model was formulated; at TU Dortmund (applicant 2), the NMPC-based RL framework was developed. So far, our approach assumes that a perfect rigorous model is available. Real chemical processes are subject to uncertainty resulting from the presence of model parameters, which are fitted to experimental data. When using surrogates, we must also account for the differences between the original model and the surrogate. Moreover, chemical engineering hardly operates single processing units, but connects them to more complex flowsheets, which may also contain recycles for mass or energy integration. This second phase of the project addresses these challenges systematically in the following way. To improve the parameter estimation, the objective of the NMPC-based RL framework is changed to an information metric based on optimal experimental design. Then, parametric uncertainty in the model parameters is quantified via Bayesian inversion or, alternatively, bootstrapping. The mismatch between rigorous and surrogate model is addressed by training a Bayesian last layer network. The quantified uncertainty is used to formulate a robust NMPC, which is adapted via RL with a rigorous model. We also investigate whether surrogate models of individual units can be combined by accounting for larger uncertainties in the individual surrogates. We apply the proposed framework on the hybrid separation process batch distillation / pervaporation to separate ethanol and water. To this end, the batch column of the first phase is extended by the membrane process and a dynamic model of the pervaporation process is built. The goal of the second phase is the optimal control of this hybrid process via our NMPC-based RL in real-time.

• DFG Programme: Priority Programmes
• Subproject of: SPP 2331: Machine Learning in Chemical Engineering. Knowledge Meets Data: Interpretability, Extrapolation, Reliability, Trust