Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning

By Sebastian Farquhar, Michael A. Osborne, Yarin Gal. 2019

72 citations recorded by the source corpus.

We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models while maintaining a distribution over weight-space with full support. Other scalable Bayesian deep learning methods, like MC dropout or deep ensembles, have discrete support-they assign zero probability to almost all of the weight-space. Unlike these discrete support methods, Radial BNNs' full support makes them suitable for use as a prior for sequential inference. In addition, they solve the conceptual challenges with the a priori implausibility of weight distributions with discrete support. The Radial BNN is motivated by avoiding a sampling problem in 'mean-field' variational inference (MFVI) caused by the so-called 'soap-bubble' pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to a challenging real-world medical application without needing ad-hoc tweaks and intensive tuning. In fact, in this setting Radial BNNs out-perform discrete-support methods like MC dropout. Lastly, by using Radial BNNs as a theoretically principled, robust alternative to MFVI we make significant strides in a Bayesian continual learning evaluation.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Posterior Meta-Replay for Continual Learning (2021, methodology): “As an alternative method to obtain explicit posterior approximations we also experimented with radial posteriors [13], which we briefly describe below.”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, background): “, 2015) and Radial BNNs (Farquhar et al., 2020) are a promising direction but have not yet lead to stable training on large image datasets.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “[372] introduced a variational approximate posterior for BNNs termed as Radial BNNs which scales well to large models.”
Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, methodology): “Mean-ﬁeld variational inference methods, such as Bayes by Back-prop (Blundell et al., 2015) and Radial BNNs (Farquhar et al., 2020) are a promising direction but have not yet lead to stable training on large image datasets.”
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2020, methodology): “These methods can be broadly classified into two categories: Bayesian approaches [5, 20, 32, 51, 57, 63, 69, 92, 118, 184, 194] and Non-Bayesian, or frequentist, approaches [2, 98, 102, 107, 111, 177].”