Subspace Inference for Bayesian Deep Learning

By Pavel Izmailov, Wesley J. Maddox, Polina Kirichenko, Timur Garipov, Dmitry P. Vetrov, Andrew Gordon Wilson. 2019

164 citations recorded by the source corpus.

Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this paper, we construct low-dimensional subspaces of parameter space, such as the first principal components of the stochastic gradient descent (SGD) trajectory, which contain diverse sets of high performing models. In these subspaces, we are able to apply elliptical slice sampling and variational inference, which struggle in the full parameter space. We show that Bayesian model averaging over the induced posterior in these subspaces produces accurate predictions and well calibrated predictive uncertainty for both regression and image classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “[19] show promising results from directly constructing subspaces for Bayesian inference.”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, methodology): “Instead of end-to-end training like rank-1 BNNs, it uses two stages where one ﬁrst performs stochastic weight averaging and then applies PCA to form a projection matrix from the set of weights to, e.g., 5 dimensions, which one can then perform inference over.”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “Not only are the likelihoods diffuse, but different settings of the parameters correspond to a diverse variety of compelling hypotheses for the data (Garipov et al., 2018; Izmailov et al., 2019).”
What Are Bayesian Neural Network Posteriors Really Like? (2021, methodology): “The Bayesian deep learning community has designed multiple successful practical methods inspired by the Bayesian approach (Blundell et al., 2015; Gal & Ghahramani, 2016; Welling & Teh, 2011; Kirkpatrick et al., 2017; Maddox et al., 2019; Izmailov et al., 2019; Daxberger et al., 2020), with applicati”
Laplace Redux - Effortless Bayesian Deep Learning (2021, background): “(a) We find a MAP estimate (yellow star) via standard training (background contours = log-posterior landscape on the two-dimensional PCA subspace of the SGD trajectory [30]).”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, background): “…can be drawn, which include probabilistic programs (Tran et al., 2017; Bingham et al., 2018), Bayesian neural networks (Neal, 1996; Saatci & Wilson, 2017; Maddox et al., 2019; Izmailov et al., 2019), and more general types of Gaussian processes (Damianou & Lawrence, 2013; Gardner et al., 2018).”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “Alternatively, one could exploit the fact that weights of a network are effectively lower dimensional and attempt to parametrize the network in this lower dimensional space (Pradier et al., 2018; Izmailov et al., 2019).”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “[241] designed low-dimensional subspaces of parameter space that comprised of diverse sets of high performing approaches.”
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, background): “This observation helps explain the success of subspace based techniques that examine the loss in a lower dimensional space such as Izmailov et al. (2019).”