Validated computation of patterns in recurrent neural networks

• Applicant: Elena Queirolo, Ph.D.
• Subject Area: Mathematics
• Term: from 2022 to 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 493789610

The goal of this project is to employ validated computer-assisted techniques from dynamical systems to understand the behaviour of recurrent neural networks (RNNs). RNNs are one key component in recent machine learning algorithms, particularly in the context of deep neural networks. RNNs can be interpreted as dynamical systems on graphs, or networks. Nodes and edges are associated to state values and weights respectively. First, information processing by the nodes of a neural net with fixed link weights is a dynamical process as initial conditions from the input layer are processed and yield, via a finite-time iteration of maps or a finite-time flow, the values at the output layer. Second, also the learning phase of a neural network can be interpreted as a dynamical minimization problem obtained by iteration. Of course, there is a feedback between the dynamics on and of the network, i.e., on the nodes and of the links. At the end of the dynamical process obtained by processing a sufficient amount of training data, one hopes that the dynamical system has produced a sufficiently stable pattern of the link weights, which has sufficient expressive power for tasks such as pattern matching or even extrapolation beyond initial training data.We are interested in studying the stable network configuration, that is the pattern, that the weights will achieve. Such patterns are of great interest because they indicate when the RNN received sufficient training, and if the training objective has been achieved. This study would also be used to compare flows with different starting conditions and flows in RNNs with different architectures. This would allow us to choose the best combination of RNN and system architecture for the application under consideration.Applying concepts from validated numerics, it will be possible to find patterns and prove their existence and stability. This is going to allow us to mathematically determine their occurrence in a given RNN. Hence, this project centers around the following question:Can we rigorously predict the pattern of a given RNN via validated numerical dynamics techniques?This question is then divided into two sub-questions: what patterns are possible in a given RNN? and which of them is stable? The main ingredient towards this goal is the use of validated numerics outside of its traditional environment. The central theme of validated numerics is to turn numerical computations into proofs. This is usually achieved by considering a numerical approximation and rigorously constructing a posteriori bounds of its error, proving the existence of a solution in the same sweep. This project requires a background in network based dynamical systems and in validated numerics, both being in the applicant's core technical capabilities.

• DFG Programme: WBP Position