Bayesian Deep Learning and a Probabilistic Perspective of Generalization

By Andrew Gordon Wilson, Pavel Izmailov. 2020

816 citations recorded by the source corpus.

The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights. Bayesian marginalization can particularly improve the accuracy and calibration of modern deep neural networks, which are typically underspecified by the data, and can represent many compelling but different solutions. We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead. We also investigate the prior over functions implied by a vague distribution over neural network weights, explaining the generalization properties of such models from a probabilistic perspective. From this perspective, we explain results that have been presented as mysterious and distinct to neural network generalization, such as the ability to fit images with random labels, and show that these results can be reproduced with Gaussian processes. We also show that Bayesian model averaging alleviates double descent, resulting in monotonic performance improvements with increased flexibility. Finally, we provide a Bayesian perspective on tempering for calibrating predictive distributions.

Subfields: Artificial Intelligence, Scaling & Learning Theory, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

In Probabilistic & Bayesian Methods

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Importantly, this suggests a departure from a popular strategy of improving iid performance of predictors by marginalizing across F∗ (Wilson and Izmailov, 2020).”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, methodology): “[41] Andrew Gordon Wilson and Pavel Izmailov.”
Posterior Meta-Replay for Continual Learning (2021, background): “…diversity in function space enables uncertainty-based OOD detection and because different architectures induce different priors in function space [77], one can expect that prior and architecture play a key role as well, which we observe by comparing the TInfer performance for different…”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, neutral): “In practice, tempering the hyperparameter distribution may prove helpful under misspecified models (Wenzel et al., 2020; Wilson & Izmailov, 2020), which is equivalent to scaling KL [qt(✓) || qt 1(✓)] by a positive scalar .”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, neutral): “It is a well-known fact that regularization [19] and ensembling [9, 20, 49] can mitigate the nonlinear peak.”
Why Normalizing Flows Fail to Detect Out-of-Distribution Data (2020, background): “In this paper, we show how the inductive biases [28, 47] of flow models — implicit assumptions in the architectures and training procedures — can hinder OOD detection.”
A survey of uncertainty in deep neural networks (2021, methodology): “(19) Wilson and Izmailov [16] argue that a key advantage of BNNs lie in this marginalization step, which particularly can improve both the accuracy and calibration of modern deep neural networks.”
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2020, background): “We see this in the leftmost plot from Figure 2: the predictions from different ensemble elements can be interpreted as samples which, when combined, approximate the predictive posterior [186].”
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, background): “…been shown to improve the generalization performance of DNNs (Hern ´ andez-Lobato & Adams, 2015; Blundell et al., 2015; Li et al., 2016a; Maddox et al., 2019; Wilson & Izmailov, 2020), while providing a principled representation of uncertainty on predictions, which is crucial for decision making.”
FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing (2020, methodology): “This procedure can be interpreted as model averaging, and it has been shown that such Bayesian model averaging can alleviate even prominent double descent behavior [88].”
Appendix for: How Good is the Bayes Posterior in Deep Neural Networks Really? (2020, neutral): “(58) Therefore, when the observation model is transformed as in (55) and as suggested by (Wilson & Izmailov, 2020), then in order to obtain a normalized observation model we must include the correction term (58) and this produces a modified energy function, ŨT (θ), that differs from the actual…”
Laplace Redux - Effortless Bayesian Deep Learning (2021, background): “Finally, note that both the LA as well as SWAG can be extended to ensembles of modes in a post-hoc manner [84, 81].”
What Are Bayesian Neural Network Posteriors Really Like? (2021, methodology): “…et al., 2020), expectation propagation (Hernández-Lobato & Adams, 2015), and leveraging the stochastic gradient descent (SGD) trajectory, either for a deterministic approximation, or sampling as in SGLD (Mandt et al., 2017; Maddox et al., 2019; Izmailov et al., 2018; Wilson & Izmailov, 2020).”
Deterministic Neural Networks with Appropriate Inductive Biases Capture Epistemic and Aleatoric Uncertainty (2021, background): “It is interesting to note that ensembling can also be seen as performing Bayesian model averaging [Wilson and Izmailov, 2020], as each ensemble member, producing a softmax output p ( y | x , ω ) , can be considered to be drawn after training from some distribution p ( ω | D ) over the model…”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, background): “More generally, Bayesian neural networks can be seen as ensembles with members being weighted by the (approximated) posterior distribution over the parameters [34, 51, 56, 7, 71, 72].”