Probabilistic Numerics - Probabilistic Programming for Autonomous Systems

• Applicant: Professor Dr. Philipp Hennig
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Mathematics
• Term: from 2015 to 2017
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 265399621

• Final Report Year: 2017

Final Report Abstract

The ambition of this Emmy Noether project was a new conceptual framework to describe numerical mathematical algorithms as autonomous machines acting rationally to manipulate an internal notion of uncertainty (a probabilistic interpretation). The funding period ended ahead of schedule, but we have already been able to make significant progress towards the objective. To this date, the work of the Emmy Noether research group has yielded probabilistic numerical analysis for a range of algorithms across the spectrum of numerical tasks: linear algebra, nonlinear optimization, and the solution of differential equations. The latter area has been particularly fruitful with respect to theory. Here, an entire new analytic framework is emerging, providing a class of solvers for differential equations that inherit both the classic method’s computational complexity and their high precision point estimates, but supplement this point estimate with a surrounding probability measure whose concentration rate matches the empirical worst case error, thus can be interpreted meaningfully as a notion of uncertainty (in the other domains, we have reached analogous, if not always so extensive, results). The corresponding publications have established a quantitative connection between classic numerical point estimates and probabilistic inference, provided a meaningful probabilistic convergence analysis showing that structured, calibrated notions of uncertainty can be constructed at acceptable or even negligible computational cost. On the practical side, the probabilistic viewpoint opens opportunities to address challenges in contemporary machine learning. As a concrete example, a probabilistic replacement for the classic line search paradigm in nonlinear optimization allows local adaptation of the learning rate in deep learning, addressing a prominent algorithmic problem in this exceptionally popular area of research. The strong industrial interest in this, still comparably simple, example of probabilistic functionality makes us hopeful for the impact of more elaborate practical work currently in preparation. An soft goal outlined in the original proposal was the development of a nascent research community in probabilistic numerics, with a leading role for the Emmy Noether research group. Several workshops and seminars were organized to this end. Partly as a result of such work, several working groups on probabilistic numerics, both informal and formal, have been formed at several institutions worldwide: The subject is among the identified key research goals of the newly formed Alan Turing Institute in London; the NSF-funded Statistical and Applied Mathematical Sciences Institute (SAMSI) in North Carolina formed a formal, multi-year working group on the subject (of which the PI is an associated member). The 15th international Dobbiaco Summer School will be focused entirely on probabilistic numerics. And the PI is currently writing a full-scale textbook on the subject under contract for Cambridge University Press, in collaboration with two colleagues in the UK. Twenty months into the project, the concept of probabilistic numerics is beginning to take hold internationally. The mathematical foundations of the concept have been shown to be solid, and the conceptional difference to other approaches have been clearly delineated. Probabilistic numerics has been shown to allow for new kinds of analysis of existing methods, and the development of novel functionality. Less formally but no less importantly, by adopting the probabilistic modeling language ubiquitous in statistics and machine learning, probabilistic computation empowers a large community of researchers from these fields to understand and extend the existing body of literature in numerical analysis to their own ends, and thus affords renewed relevance to numerical analysis in these lively communities.

Publications

• A Random Riemannian Metric for Probabilistic Shortest-Path Tractography in Navab, Hornegger, Wells & Frangi, eds.; Medical Image Computing and Computer Assisted Intervention (MICCAI) vol. 18 (2015), Springer LNCS vol. 9349, pp. 597–604
  Søren Hauberg, Michael Schober, Matthew Liptrot, Philipp Hennig, Aasa Feragen
  
• Probabilistic Interpretation of Linear Solvers. SIAM Journal on Optimization (SIOPT) vol. 25 no. 1 (2015), pp. 234–260
  Philipp Hennig
  
• Probabilistic Line Searches for Stochastic Optimization, in Cortes, Lawrence, Lee, Sugiyama & Garnett, eds.; Advances in Neural Information Processing Systems (NIPS) vol. 28 (2015), pp. 181–189
  Maren Mahsereci, Philipp Hennig
  
• Probabilistic Numerics and Uncertainty in Computations. Proceedings of the Royal Society A, vol. 471 nr. 2179 (2015)
  Philipp Hennig, Michael A. Osborne, Mark Girolami
  
• A probabilistic model for the numerical solution of initial value problems
  Michael Schober, Simo Särkkä, Philipp Hennig
  
• Active Uncertainty Calibration in Bayesian ODE Solvers, in Ihler & Janzing, eds.; Uncertainty in Artificial Intelligence (UAI), vol. 32 (2016), pp. 309–318
  Hans Kersting & Philipp Hennig
  
• Probabilistic Approximate Least-Squares, in Gretton & Robert, eds.; Artificial Intelligence and Statistics (AISTATS) vol. 19 (2016); Journal of Machine Learning Research W&CP vol. 51, pp. 676–684
  Simon Bartels & Philipp Hennig