Dynamical systems approach to robust reconstruction of probability distributions of observed data

• Applicant: Privatdozent Dr. Pavel Gurevich, Ph.D.
• Subject Area: Mathematics ; Statistics and Econometrics ; Theoretical Computer Science
• Term: from 2018 to 2020
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 415860776

• Final Report Year: 2022

Final Report Abstract

The project dealt with learning probability distributions of observed data by artificial neural networks. We suggested a so-called gradient conjugate prior (GCP) update appropriate for neural networks, which is a modification of the classical Bayesian update for conjugate priors. We established a connection between the gradient conjugate prior update and the maximization of the log-likelihood of the predictive distribution. Further, we showed that contaminating the training data set by outliers leads to bifurcation of a stable equilibrium from infinity. Using the outputs of the GCP network at the equilibrium, we derived an explicit formula for correcting the learned variance of the marginal distribution and for removing the bias caused by outliers in the training set. Assuming a Gaussian (input-dependent) ground truth distribution contaminated with a proportion ε of outliers, we showed that the fitted mean is in a ce1/ε -neighborhood of the ground truth mean and the corrected variance is in a bε-neighborhood of the ground truth variance, whereas the uncorrected variance of the marginal distribution can even be infinite. We explicitly found b as a function of the output of the GCP network, without a priori knowledge of the outliers (possibly input-dependent) distribution. Experiments with synthetic and real-world data sets indicate that the GCP network fitted with a standard optimizer outperforms other robust methods for regression.

Publications

• Pairing an arbitrary regressor with an artificial neural network estimating aleatoric uncertainty. Neurocomputing 350 (2019), 291–306
  Gurevich P., Stuke H.
  (See online at https://doi.org/10.1016/j.neucom.2019.03.031)
• Gradient conjugate priors and deep neural networks. Artificial Intelligence 278 (2020), 103184
  Gurevich P., Stuke H.
  (See online at https://doi.org/10.1016/j.artint.2019.103184)
• Robustness against outliers for deep neural networks by gradient conjugate priors. SIAM J. Appl. Dyn. Syst., 19(4), 2567–2593
  Gurevich P., Stuke H.
  (See online at https://doi.org/10.1137/20M131727X)