Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
By Yarin Gal, Zoubin Ghahramani. 2015
12,300 citations recorded by the source corpus.
Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.
Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Percy Liang
- Quoc V. Le
- Jianfeng Gao
- Ruslan Salakhutdinov
- Pieter Abbeel
In Probabilistic & Bayesian Methods
- Yarin Gal
- Richard E. Turner
- Zoubin Ghahramani
- Stefano Ermon
- Max Welling
- Yoshua Bengio
- José Miguel Hernández-Lobato
- David M. Blei
Citations in the corpus
- Using the Output Embedding to Improve Language Models (2016, methodology): “Following that, (Zaremba et al., 2014) introduced a dropout (Srivastava, 2013) augmented NNLM. (Gal, 2015; Gal and Ghahramani, 2016) proposed a new dropout method, which is referred to as Bayesian Dropout below, that improves on the results of (Zaremba et al., 2014).”
- Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “Many successful approaches have utilized a classifier’s predictive distribution for OOD detection (Gal & Ghahramani, 2016; Wang et al., 2018; Liang et al., 2017).”
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, methodology): “drawn from a Gaussian prior, W ∼ N(0, I), and using variational inference to estimate the posterior distribution p(W |X,Y ) [17].”
- Scalable agent alignment via reward modeling: a research direction (2018, methodology): “A number of recent works develop scaleable approximate Bayesian methods for neural networks, beginning with Graves (2011), Blundell et al. (2015), Kingma et al. (2015), Hernández-Lobato & Adams (2015), and Gal & Ghahramani (2016).”
- Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “Dropout (Gal and Ghahramani, 2016) is applied with probability p = 0.1 to the embedding matrices, with p = 0.2 to the input and recurrent connections of the RNNs.”
- Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “We leverage the fact that a network trained with dropout can be used to approximate a Bayesian uncertainty estimate of the network’s output (Gal and Ghahramani, 2016).”
- Concrete Problems in AI Safety (2016, background): “…in that the former involves attacks against a legitimate ML system by an adversary (e.g. a criminal tries to fool a face recognition system), while the latter involves attacks by an ML system controlled by an adversary (e.g. a criminal trains a “smart hacker” system to break into a website).”
- Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “A large amount of research has been dedicated towards calibrating the model posterior, pθ(ŷn+1|xn+1), such that the accuracy, yn+1 = ŷn+1, is indeed equal to the estimated probability (Niculescu-Mizil and Caruana, 2005; Gal and Ghahramani, 2016; Guo et al., 2017).”
- Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, methodology): “From a Bayesian perspective, dropout is a method to introduce prior distribution for the parameters rather than a regularization that introduces perturbation to the inputs and the hidden variables [6], [26].”
- A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, methodology): “(3) can be approximated as L2 regularisation over the variational parameters m k [17].”
- Temporal Ensembling for Semi-Supervised Learning (2016, neutral): “In bootstrap aggregating, or bagging, multiple networks are trained independently based on subsets of training data (Breiman, 1996).”
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “the number of nonzero network parameters. Narang et al. integrate magnitudebased pruning into training [40]. Gal and Ghahramani show that dropout approximates Bayesian inference in Gaussian processes [10]. Bayesian perspectives on dropout learn the dropout probabilities during training [11, 28, 46].”
- Making Neural QA as Simple as Possible but not Simpler (2017, methodology): “Throughout all experiments we use a hidden dimensionality of n = 300 , variational dropout at the input embeddings with the same mask for all words (Gal and Ghahramani, 2015) and a rate of 0 .”
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “Recently, Gal and Ghahramani (2016) proposed using Monte Carlo dropout (MC-dropout) to estimate predictive uncertainty by using Dropout (Srivastava et al., 2014) at test time.”
- Evidential Deep Learning to Quantify Classification Uncertainty (2018, methodology): “Improvement of the approximation techniques, such as Variational Bayes (VB) [2, 6, 27, 23, 9] and Stochastic Gradient Hamiltonian Monte Carlo (SG-HMC) [3] tailored specifically for scalable inference of BNNs is an active research field.”
- A General Framework for Uncertainty Estimation in Deep Learning (2019, methodology): “The larger the number of samples, the better the estimates [3].”
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “…generalization may find use in offline RL algorithms, including techniques from causal inference (Schölkopf, 2019), uncertainty estimation (Gal and Ghahramani, 2016; Kendall and Gal, 2017), density estimation and generative modeling (Kingma et al., 2014), distributional robustness (Sinha…”