Weight Uncertainty in Neural Networks

By Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, Daan Wierstra. 2015

2,165 citations recorded by the source corpus.

We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

Subfields: Artificial Intelligence, Management Science and Operations Research, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Probabilistic & Bayesian Methods

Citations in the corpus

Overcoming catastrophic forgetting in neural networks (2016, methodology): “Our initial explorations suggest that one might improve on this local estimate by using Bayesian neural networks (37).”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “A number of recent works develop scaleable approximate Bayesian methods for neural networks, beginning with Graves (2011), Blundell et al. (2015), Kingma et al. (2015), Hernández-Lobato & Adams (2015), and Gal & Ghahramani (2016).”
Concrete Problems in AI Safety (2016, background): “Robot locomotion or industrial control tasks might be a natural candidate for such experiments.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “…work on uncertainty in neural networks has focussed mostly on relatively faster approximate Bayesian solutions such as variational Bayesian methods (Blundell et al., 2015; Graves, 2011; Louizos and Welling, 2016), assumed density ﬁltering (Hern´andez-Lobato and Adams, 2015), expectation…”
Evidential Deep Learning to Quantify Classification Uncertainty (2018, methodology): “Improvement of the approximation techniques, such as Variational Bayes (VB) [2, 6, 27, 23, 9] and Stochastic Gradient Hamiltonian Monte Carlo (SG-HMC) [3] tailored specifically for scalable inference of BNNs is an active research field.”
A General Framework for Uncertainty Estimation in Deep Learning (2019, neutral): “, exponential-family [2], [5], [8], [9].”
Deep Variational Information Bottleneck (2017, methodology): “4 Even if our choice of encoding distribution and variational prior do not admit an analytic KL, we could similarly reparameterize through a sample of the divergence (Kingma & Welling, 2014; Blundell et al., 2015).”
Decentralized Bayesian Learning over Graphs (2019, neutral): “While we compute the KL divergence in (1) in a closed form, we employ simple Monte Carlo to compute the gradients using Bayes by Backprop [10, 5].”
Peer-to-peer Federated Learning on Graphs (2019, methodology): “We use the Bayes by Back-prop training algorithm proposed by Blundell et al. (2015) to learn the Gaussian variational posterior.”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “3 Our Hypothesis We build our Bayesian Cnn upon Bayes by Backprop [19], [4].”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “While the diagonal covariance approximation is standard in Bayesian deep learning [3, 27], it can be too restrictive.”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “Hence, different degrees of approximations are made to reduce the computational overhead [40], [41], [42], [43], including about the prior that is specified for the parameters of the neural network.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “We therefore plan to extend this work to other model classes, such as Bayesian neural networks (Neal, 1996; Hernández-Lobato and Adams, 2015; Blundell et al., 2015; Springenberg et al., 2016; Klein et al., 2017), which may lower the computational overhead while having similar predictive quality.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “Whilst accurate Bayesian NN inference is possible with sufficient compute [10], approximate inference methods [45], [38], [46] have enjoyed recent popularity given their simpler implementation and faster training times.”
Probabilistic Model-Agnostic Meta-Learning (2018, background): “Although exact inference in Bayesian neural networks is impractical, approximations based on backpropagation and sampling [16, 32, 20, 2] have been effective in incorporating uncertainty into the weights of generic networks.”