Noisy Natural Gradient as Variational Inference

By Guodong Zhang, David Kristjanson Duvenaud, Roger Baker Grosse. 2017

228 citations recorded by the source corpus.

Variational Bayesian neural nets combine the flexibility of deep learning with Bayesian uncertainty estimation. Unfortunately, there is a tradeoff between cheap but simple variational families (e.g.~fully factorized) or expensive and complicated inference procedures. We show that natural gradient ascent with adaptive weight noise implicitly fits a variational posterior to maximize the evidence lower bound (ELBO). This insight allows us to train full-covariance, fully factorized, or matrix-variate Gaussian variational posteriors using noisy versions of natural gradient, Adam, and K-FAC, respectively, making it possible to scale up to modern-size ConvNets. On standard regression benchmarks, our noisy K-FAC algorithm makes better predictions and matches Hamiltonian Monte Carlo's predictive variances better than existing methods. Its improved uncertainty estimates lead to more efficient exploration in active learning, and intrinsic motivation for reinforcement learning.

Subfields: Artificial Intelligence, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “Ritter et al. [54] proposed the use of either a diagonal or block Kronecker factored (KFAC) approximation to the Hessian matrix for Laplace approximations, and Ritter et al. [53] successfully applied the KFAC approach to online learning scenarios.”
Generalized Variational Continual Learning (2020, background): “As mentioned in Section 2, likelihood-tempering (or KL-reweighting) has been empirically found to improve performance when using variational inference for Bayesian Neural Networks over a wide number of contexts and papers (Osawa et al., 2019; Zhang et al., 2018).”
Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks (2020, methodology): “Many methods to achieve this goal have been proposed, both Bayesian (e.g. Blundell et al., 2015; Louizos & Welling, 2017; Zhang et al., 2018) and non-Bayesian (e.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “, 2015; Hernández-Lobato and Adams, 2015) or improving its posterior approximation (Louizos and Welling, 2016; Zhang et al., 2017; Krueger et al., 2017).”
Task Agnostic Continual Learning Using Online Variational Bayes (2018, methodology): “To do so, we followed the experiment described in [36].”
Practical Deep Learning with Bayesian Principles (2019, background): “Our work solves these issues by applying deep-learning techniques to natural-gradient VI [24, 56].”
Three Mechanisms of Weight Decay Regularization (2018, methodology): “Following on that work, K-FAC has been adopted in many tasks (Wu et al., 2017; Zhang et al., 2017) to gain optimization beneﬁts, and was shown to be amendable to distributed computation (Ba et al., 2016).”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “A recent independent work on noisy-Adam by Zhang et al. (2018) is algorithmically very similar to our Vadam method, however their derivation lacks a strong motivation for the use of momentum.”
Learning Neural Networks with Adaptive Regularization (2019, methodology): “Kronecker Factorization in Optimization The Kronecker factorization assumption has also been applied in the literature of neural networks to approximate the Fisher information matrix in secondorder optimization methods [31, 42].”
Limitations of the Empirical Fisher Approximation (2019, methodology): “The EF has been used in the context of variational inference by various authors [Graves, 2011, Zhang et al., 2018, Salas et al., 2018, Khan et al., 2018, Mishkin et al., 2018], some of which have drawn further connections between NGD and Adam.”
Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs (2018, methodology): “K-FAC has been successfully applied to convolutional neural networks [10], distributed memory training of ImageNet [5], recurrent neural networks [21], Bayesian deep learning [30], and reinforcement learning [27].”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, background): “Stochastic gradient MCMC easily fits on top of gradient-based optimizers by adding noise, as well as certain variational inference algorithms (Zhang et al., 2017; Khan et al., 2018).”
Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods (2018, methodology): “Following the standard approach in this line of work (Khan & Lin, 2017; Zhang et al., 2017; Khan et al., 2017, 2018), the Fisher Information (FI) can be used to identify the variance, P (gi(t)|w ∗ i (t)) = N (Hii (w ∗ i (t)− µi(t)) , Hii) .”
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise (2019, background): “…devoted to this in deep learning; for example, scaling the learning rate or batch-size (Smith 1This corresponds to the matrix-variate Gaussian noise in Zhang et al. (2018) and Le, 2017; Goyal et al., 2017; Hoffer et al., 2017) to augment gradient noise in order to encourage better generalization.”
A survey of uncertainty in deep neural networks (2021, background): “1[20] 3[30] 4[149] 5[75] 6[213] 7[129] 8[63] 9[80] 10[150] 11[143] 12[84] 13[235] 14[145] 15[135] 16[146] 17[272]”
Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights (2020, background): “…prior that better models the structures present in the weight a-priori [Ghosh et al., 2018, Neal, 2012, Blundell et al., 2015], and (iii) using structured variational approximations that can potentially capture weight correlations in the posterior [Louizos and Welling, 2016, Zhang et al., 2017].”
Understanding Short-Horizon Bias in Stochastic Meta-Optimization (2018, methodology): “In the setting of Bayesian neural networks, quadratic approximations to the log-likelihood motivated the Laplace approximation (MacKay, 1992) and variational inference (Graves, 2011; Zhang et al., 2017).”