Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam

By Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, Akash Srivastava. 2018

296 citations recorded by the source corpus.

Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms can be implemented within the Adam optimizer by perturbing the network weights during gradient evaluations, and uncertainty estimates can be cheaply obtained by using the vector that adapts the learning rate. This requires lower memory, computation, and implementation effort than existing VI methods, while obtaining uncertainty estimates of comparable quality. Our empirical results confirm this and further suggest that the weight-perturbation in our algorithm could be useful for exploration in reinforcement learning and stochastic optimization.

Subfields: Artificial Intelligence, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “It may also be possible to similarly utilize Adam and other stochastic ﬁrst-order methods, which view as a promising direction for future work.”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, background): “Comparing with the classical strategy that perturbs weight in the entire space [40, 16, 42, 10], we focus on characterizing the weight loss landscape on the new task with respect to the important subspace representing old task.”
Continual Deep Learning by Functional Regularisation of Memorable Past (2020, methodology): “As a consequence the variance estimates might not be very good during the iteration, even though the ﬁxed-point of the algorithm is not changed (Khan et al., 2018).”
Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “, 2018) uses a Gauss-Newton approximation introduced by (Schraudolph, 2002; Graves, 2011) to estimate the VCL objective function and used natural gradient method proposed in (Khan et al., 2018) to exploit the Riemannian geometry of the variational posterior by scaling the gradient with an adaptive l”
Approximate Inference Turns Deep Networks into Gaussian Processes (2019, methodology): “We train LeNet-5 on digits 0 and 1 with VOGN and visualize the predictive mean and kernel on all 10 classes denoted by differently colored regions on the y-axis.”
Practical Deep Learning with Bayesian Principles (2019, methodology): “Our work solves these issues by applying deep-learning techniques to natural-gradient VI [24, 56].”
Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates (2020, background): “Martens and Grosse (2015), Zhang, Sun, Duvenaud, and Grosse (2017), and Khan et al. (2018) suggested using the connection between natural gradient descent (Amari, 1998) and variational inference to perform natural gradient optimization in deep neural networks.”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “Recent work has largely focused on scalable inference (e.g., Blundell et al., 2015; Gal & Ghahramani, 2016; Kendall & Gal, 2017; Ritter et al., 2018; Khan et al., 2018; Maddox et al., 2019), function-space inspired priors (e.”
Limitations of the Empirical Fisher Approximation (2019, methodology): “The EF has been used in the context of variational inference by various authors [Graves, 2011, Zhang et al., 2018, Salas et al., 2018, Khan et al., 2018, Mishkin et al., 2018], some of which have drawn further connections between NGD and Adam.”
Advances in Variational Inference (2017, background): “Recent advances in this direction include [64], [106], [121].”
SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient (2018, methodology): “SLANG is based on a natural-gradient method that employs the empirical Fisher approximation [19].”
Natural Variational Continual Learning (2018, methodology): “The update has a very similar form to the Vadam algorithm and is very simple to implement.”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, background): “Stochastic gradient MCMC easily fits on top of gradient-based optimizers by adding noise, as well as certain variational inference algorithms (Zhang et al., 2017; Khan et al., 2018).”
A survey of uncertainty in deep neural networks (2021, methodology): “Lastly, connections of variation inference to Adam [155], RMS Prop [156] and batch normalization [157] have been”
Single-Model Uncertainties for Deep Learning (2018, methodology): “Prior literature on uncertainty estimation for deep neural networks is dominated by Bayesian methods [37, 6, 25, 44, 45, 79], implemented in approximate ways to circumvent their computational intractability.”
Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting (2020, background): “Variational Inference (VI) for Bayesian neural networks (Graves, 2011; Blundell et al., 2015; Khan et al., 2018) aims at estimating the posterior distribution of model weights given training data.”
Adversarial Weight Perturbation Helps Robust Generalization (2020, methodology): “Different from the commonly used random weight perturbation (sampling a random direction) [54, 19, 16], we propose the Adversarial Weight Perturbation (AWP), along which the adversarial loss increases dramatically.”
Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods (2018, methodology): “Following the standard approach in this line of work (Khan & Lin, 2017; Zhang et al., 2017; Khan et al., 2017, 2018), the Fisher Information (FI) can be used to identify the variance, P (gi(t)|w ∗ i (t)) = N (Hii (w ∗ i (t)− µi(t)) , Hii) .”