Appendix for: How Good is the Bayes Posterior in Deep Neural Networks Really?

By Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Bartlomiej Swiatkowski, Linh Hoang Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, Sebastian Nowozin. 2020

424 citations recorded by the source corpus.

During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there are---as of early 2020---no publicized deployments of Bayesian neural networks in industrial practice. In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD. Furthermore, we demonstrate that predictive performance is improved significantly through the use of a "cold posterior" that overcounts evidence. Such cold posteriors sharply deviate from the Bayesian paradigm but are commonly used as heuristic in Bayesian deep learning papers. We put forward several hypotheses that could explain cold posteriors and evaluate the hypotheses through experiments. Our work questions the goal of accurate posterior approximations in Bayesian deep learning: If the true Bayes posterior is poor, what is the use of more accurate approximations? Instead, we argue that it is timely to focus on understanding the origin of the improved performance of cold posteriors.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Posterior Meta-Replay for Continual Learning (2021, methodology): “A potential cause for the differences in performance can be that, as opposed to Farquhar et al. [13], we tempered the posteriors [76].”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, neutral): “In practice, tempering the hyperparameter distribution may prove helpful under misspecified models (Wenzel et al., 2020; Wilson & Izmailov, 2020), which is equivalent to scaling KL [qt(✓) || qt 1(✓)] by a positive scalar .”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, methodology): “We however disagree with the discussion in Wenzel et al. (2020) on the connection between tempered posteriors and different likelihoods, and believe it misses the point.”
Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting (2021, neutral): “Bayesian neural networks can also be used to provide epistemic uncertainty in fore-casts as well as detect distributional shifts (Zhu & Laptev, 2018), although these methods often do not perform as well empirically (Wenzel et al., 2020).”
Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation (2020, background): “That lower T ens is preferred was also observed in [46] and using T act > 1 for better calibration is noted in [19].”
Second opinion needed: communicating uncertainty in medical machine learning (2021, background): “Recent work by Wenzel et al. 45 demonstrates that fundamental unresolved challenges remain to estimating p (W | D) in a manner that improves predictive uncertainty in variational inference and Bayesian neural networks more generally.”
Laplace Redux - Effortless Bayesian Deep Learning (2021, background): “[79] Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Świątkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin.”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, background): “More generally, Bayesian neural networks can be seen as ensembles with members being weighted by the (approximated) posterior distribution over the parameters [28, 41, 45, 5, 57, 58].”
What Are Bayesian Neural Network Posteriors Really Like? (2021, methodology): “Furthermore, in Section 6, we show that Bayesian neural networks can achieve performance superior to SGD and even deep ensembles at temperature T = 1, in particular using the same ResNet-20-FRN model on CIFAR-10 and CNN-LSTM model on IMDB used by Wenzel et al. (2020).”