What Are Bayesian Neural Network Posteriors Really Like?

By Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, Andrew Gordon Wilson. International Conference on Machine Learning, 2021

474 citations recorded by the source corpus.

The posterior over Bayesian neural network (BNN) parameters is extremely high-dimensional and non-convex. For computational reasons, researchers approximate this posterior using inexpensive mini-batch methods such as mean-field variational inference or stochastic-gradient Markov chain Monte Carlo (SGMCMC). To investigate foundational questions in Bayesian deep learning, we instead use full-batch Hamiltonian Monte Carlo (HMC) on modern architectures. We show that (1) BNNs can achieve significant performance gains over standard training and deep ensembles; (2) a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains; (3) in contrast to recent studies, we find posterior tempering is not needed for near-optimal performance, with little evidence for a"cold posterior"effect, which we show is largely an artifact of data augmentation; (4) BMA performance is robust to the choice of prior scale, and relatively similar for diagonal Gaussian, mixture of Gaussian, and logistic priors; (5) Bayesian neural networks show surprisingly poor generalization under domain shift; (6) while cheaper alternatives such as deep ensembles and SGMCMC methods can provide good generalization, they provide distinct predictive distributions from HMC. Notably, deep ensemble predictive distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/26190ccd4753d6c0d25499ea34475205ebdd8240

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “In our follow-up work (Izmailov et al., 2021), we show that there is no cold posterior effect in any of the examples of Wenzel et al.”
Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors (2020, background): “Such pathology commonly arises with mixtures (Stephens, 2000), hierarchical models (Liu and Hodges, 2003), and overparametrized models (Izmailov et al., 2021).”
Transformers Can Do Bayesian Inference (2021, methodology): “Recently, posterior approximation methods, such as stochastic variational inference (SVI) and stochastic gradient MCMC methods (such as NUTS) allow for much faster convergence, which has sparked a lot of interest in the area (Izmailov et al., 2021; Fortuin et al., 2021).”
Laplace Redux - Effortless Bayesian Deep Learning (2021, background): “[80] Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Wilson.”
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users (2020, methodology): “This is an example of statistical unidentiﬁability, which can lead to complex multimodal posteriors that are hard to sample and approximate when training a BNN [22].”