Practical Deep Learning with Bayesian Principles

By Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, Rio Yokota, Mohammad Emtiyaz Khan. 2019

280 citations recorded by the source corpus.

Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch normalisation, data augmentation, and distributed training, we achieve similar performance in about the same number of epochs as the Adam optimiser, even on large datasets such as ImageNet. Importantly, the benefits of Bayesian principles are preserved: predictive probabilities are well-calibrated, uncertainties on out-of-distribution data are improved, and continual-learning performance is boosted. This work enables practical deep learning while preserving benefits of Bayesian principles. A PyTorch implementation is available as a plug-and-play optimiser.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “In the spirit of Bhatt et al. (2020a), we desire counterfactuals that are 1) informative : they should highlight features which affect our BNN’s uncertainty, and 2) relevant : counterfactuals should lie close to the original inputs and represent plausible parameter settings, lying close to the data ”
Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “Recent work has also combined Bayesian continual learning with natural gradient descent [32, 43], and in this case a relatively high value of λ = 100 was similarly found to maximize performance [32].”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “The non-stationary learning problem can be alleviated by understanding and inferring causal relations from data (e.g. Osawa et al., 2019) – which is an outstanding challenge (Pearl, 2009) – and coming up with combinations that are unlikely to be present in training distributions (Bengio, 2019).”
Generalized Variational Continual Learning (2020, background): “Examples of this are seen in Zhang et al. (2018) and Osawa et al. (2019), where the latter uses a “data augmentation factor” for down-weighting.”
Continual Deep Learning by Functional Regularisation of Memorable Past (2020, methodology): “We also run FROMP on a smaller network architecture (Swaroop et al., 2019), obtaining 99 .”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, background): “Variational Online Gauss Newton reports results on ResNet-18, outperforming a deterministic baseline in terms of NLL but not accuracy, and using 2x the number of neural network weights (Osawa et al., 2019).”
Approximate Inference Turns Deep Networks into Gaussian Processes (2019, methodology): “A similar algorithm has recently been used in [19] where it shows competitive performance to Adam and SGD.”
Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “Further exploration takes natural-gradient variational inference and enables practical and affordable modern deep learning training while preserving the beneﬁts of Bayesian principles (Osawa et al., 2019).”
Laplace Redux - Effortless Bayesian Deep Learning (2021, background): “Also, their optimization process is slower and typically more unstable unless carefully tuned [13].”
Organizing recurrent network dynamics by task-computation to enable continual learning (2020, background): “Directions in parameter space that are well explored on previous tasks are modiﬁed more slowly, hinting towards a potential connection with uncertainty-based approaches towards adaptive learning rates [35].”
Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection (2019, background): “Alternatively, one can use predictive uncertainty estimates for OoD detection (Malinin & Gales, 2018; Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Osawa et al., 2019; Ovadia et al., 2019).”
Hierarchical Indian buffet neural networks for Bayesian continual learning (2019, methodology): “This could be performed with other inference methods such as using natural gradients [Osawa et al., 2019] or only training certain parts of the convolutional block of the CNN [Ovadia et al.”
Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning (2019, methodology): “Instead of sampling the posterior distribution directly, we use the local reparameterization trick Rezende et al. [2014], Kingma et al.”
A survey of uncertainty in deep neural networks (2021, methodology): “1[20] 3[30] 4[149] 5[75] 6[213] 7[129] 8[63] 9[80] 10[150] 11[143] 12[84] 13[235] 14[145] 15[135] 16[146] 17[272]”
A Unified and General Framework for Continual Learning (2024, methodology): “Natural Gradient CL as A Special Case Natural Gradient CL (Osawa et al., 2019; Kao et al., 2021) (NCL) is a Bayesian-based CL method.”
Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning (2021, background): “(Gal & Ghahramani, 2016a; Gal, 2016; Osawa et al., 2019), often measured by the uncertainty of the model.”
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users (2020, methodology): “es-by-backprop, e.g., to use the Adam optimizer [41] instead of stochastic gradient descent. Natural gradient descent has also been advocated as an optimization tool for variational inference for BNN [65], as the distributions from the exponential family lead to, if parametrized in natural space, a ”