Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks

By José Miguel Hernández-Lobato, Ryan P. Adams. 2015

1,024 citations recorded by the source corpus.

Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, methodology): “A number of recent works develop scaleable approximate Bayesian methods for neural networks, beginning with Graves (2011), Blundell et al. (2015), Kingma et al. (2015), Hernández-Lobato & Adams (2015), and Gal & Ghahramani (2016).”
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, background): “Further, they require more time to converge and do not improve on existing techniques.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “…Bayesian solutions such as variational Bayesian methods (Blundell et al., 2015; Graves, 2011; Louizos and Welling, 2016), assumed density ﬁltering (Hern´andez-Lobato and Adams, 2015), expectation propagation (Hasenclever et al., 2015; Li et al., 2015) or stochastic gradient Langevin diffusion…”
A General Framework for Uncertainty Estimation in Deep Learning (2019, methodology): “However, these approaches are particularly dif-ﬁcult to train [2] and are rarely used in robotic applications.”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, methodology): “We considered two different UCI (Lichman, 2013) regression datasets, Boston housing and protein structure as described by Hernández-Lobato & Adams (2015) and report the negative log-likelihood of the validation data.”
Variational Continual Learning (2017, background): “, 2000), EP (Hernández-Lobato and Adams, 2015), and approximate Power EP (Hernández-Lobato et al.”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, methodology): “While we primarily focus on image classification, we show that SWAG can significantly improve test perplexities of LSTM networks on language modeling problems, and in Appendix 7 we also compare SWAG with Probabilistic Back-propagation (PBP) [16], Deterministic Variational Inference (DVI) [47], and D”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, methodology): “Several approaches have been proposed based on, e.g., the Laplace approximation (MacKay, 1992b), Hamiltonian Monte Carlo (Neal, 1995), expectation propagation (Jylänki et al., 2014; Hernández-Lobato & Adams, 2015), and variational inference (Hinton & Van Camp, 1993; Graves, 2011).”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “We therefore plan to extend this work to other model classes, such as Bayesian neural networks (Neal, 1996; Hernández-Lobato and Adams, 2015; Blundell et al., 2015; Springenberg et al., 2016; Klein et al., 2017), which may lower the computational overhead while having similar predictive quality.”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “Fortunately, there is a rich literature concerning approximations to these objects [1; 7; 8].”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “Whilst accurate Bayesian NN inference is possible with sufficient compute [10], approximate inference methods [45], [38], [46] have enjoyed recent popularity given their simpler implementation and faster training times.”
Bayesian Hypernetworks (2017, methodology): “, 2015), expectation propagation (Hernandez-Lobato and Adams, 2015; Soudry et al., 2014), or α-divergences (Li and Gal, 2017).”
Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “Several approaches have been proposed for Bayesian neural networks, based on, e.g., the Laplace approximation (MacKay, 1992a), Hamiltonian Monte Carlo (Neal, 2012), variational inference (Hinton & Van Camp, 1993; Graves, 2011), and probabilistic backpropagation (Hern ´ andez-Lobato & Adams, 2015).”
Uncertainty-based Continual Learning with Adaptive Regularization (2019, methodology): “In addition, there are several practical methods for variational inference in neural networks, such as using dropout [5] or Expectation-Propagation [8].”
Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017, background): “Many approximations have been proposed over the years (Denker & LeCun, 1991; Neal, 1995; Graves, 2011; Blundell et al., 2015; Hernandez-Lobato & Adams, 2015; Hern´andez-Lobato et al., 2016), some more practical and some less practical.”
Variational Dropout and the Local Reparameterization Trick (2015, background): “Related is also probabilistic backpropagation (Hern´andez-Lobato and Adams, 2015), an algorithm for inferring marginal posterior probabilities; however, it requires certain tractabilities in the network making it insuitable for the type of models under consideration in this paper.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “Most recent work in Bayesian neural networks has been in variational inference, either speeding up its training (Graves, 2011; Blundell et al., 2015; Hernández-Lobato and Adams, 2015) or improving its posterior approximation (Louizos and Welling, 2016; Zhang et al.”
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling (2018, background): “Although promising, we did not include expectation propagation [24] and α-divergence minimization of which variational inference is a special case [25].”