Dropout Inference in Bayesian Neural Networks with Alpha-divergences

By Yingzhen Li, Yarin Gal. 2017

215 citations recorded by the source corpus.

To obtain uncertainty estimates with real-world Bayesian deep learning models, practical inference approximations are needed. Dropout variational inference (VI) for example has been used for machine vision and medical applications, but VI can severely underestimates model uncertainty. Alpha-divergences are alternative divergences to VI's KL objective, which are able to avoid VI's uncertainty underestimation. But these are hard to use in practice: existing techniques can only use Gaussian approximating distributions, and require existing models to be changed radically, thus are of limited use for practitioners. We propose a reparametrisation of the alpha-divergence objectives, deriving a simple inference technique which, together with dropout, can be easily implemented with existing models by simply changing the loss of the model. We demonstrate improved uncertainty estimates and accuracy compared to VI in dropout networks. We study our model's epistemic uncertainty far away from the data using adversarial images, showing that these can be distinguished from non-adversarial images by examining our model's uncertainty.

Subfields: Artificial Intelligence, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Evidential Deep Learning to Quantify Classification Uncertainty (2018, methodology): “Improvement of the approximation techniques, such as Variational Bayes (VB) [2, 6, 27, 23, 9] and Stochastic Gradient Hamiltonian Monte Carlo (SG-HMC) [3] tailored specifically for scalable inference of BNNs is an active research field.”
PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples (2017, background): “In order to catch any kind of threat, existing work has utilized confidence estimates from Bayesian neural networks (BNNs) or dropout (Li & Gal, 2017; Feinman et al., 2017).”
An overview of deep learning in medical imaging focusing on MRI (2018, background): “The ideas date back to the early 90s [351–353], but the field has recently seen renewed interest from the machine learning community at large, as new ways of computing uncertainty estimates from state of the art deep learning models have been developed [58,64,354].”
Bayesian Hypernetworks (2017, methodology): “…learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal and Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato and Adams, 2015; Soudry et al., 2014), or α -divergences (Li and Gal, 2017).”
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (2017, methodology): “As another line of research, Bayesian probabilistic models [10, 12] and ensembles [9] were also investigated.”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, methodology): “To do this we follow Li and Gal [37] by plotting the performance of the classifiers’s error rate and entropies on the MNIST test set as the magnitude of the perturbation added is increased.”
Concrete Dropout (2017, background): “From the detection of adversarial examples [25], through an agent exploring its environment safely [10, 18], to analysing failure cases in autonomous driving vision systems [20].”
SoK: Security and Privacy in Machine Learning (2018, background): “when predicting far from their training subspace [87].”
Uncertainty Estimation in Bayesian Neural Networks And Links to Interpretability (2018, methodology): “Our model architecture is identical to that of [42], containing two 2D-convolutional layers of 32 filters and a 3×3 kernel with ReLU activation, followed by a 2×2 max-pooling layer, and two Dense layers with dropout.”
Boosting Adversarial Attacks with Momentum (2017, background): “Among many attempts [13, 3, 15, 10, 24, 17, 11], adversarial training is the most extensively investigated way to increase the robustness of DNNs [5, 10, 24].”
Hierarchical Novelty Detection for Visual Object Recognition (2018, unknown): “There have been recent efforts toward developing efficient novelty detection methods [2, 13, 17, 20, 25], but most of the existing methods measure only the model uncertainty, i.”
A Scalable Laplace Approximation for Neural Networks (2018, methodology): “Furthermore, we perform a targeted attack that attempts to force the network to predict a specific class, in our case ‘0’ following (Li & Gal, 2017).”
Adversarial Phenomenon in the Eyes of Bayesian Deep Learning (2017, background): “Similarly, [20] and [22] examine epistemic uncertainty in the form of predictive entropy for adversarial images.”
Conditional Density Estimation with Bayesian Normalising Flows (2018, background): “Additionally, they yield state of the art performance on two of these datasets, ‘energy’ and ‘yacht’ (Hern ´ andez-Lobato & Adams, 2015; Bui et al., 2016; Li & Gal, 2017; Louizos & Welling, 2016).”
Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems (2018, methodology): “Recently, Gal et al. developed a new way to quantify uncertainty in DNNs by using dropout [38, 39, 40], which is largely used as a regularization technique [41, 42] to address the problem of over-ﬁtting.”
Variational Implicit Processes (2018, methodology): “mean-field Gaussian approximation to the posterior of Bayesian neural network weights [10,42,59], also takesO(CMS) time.”