Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models
By Mohammad Emtiyaz Khan, Didrik Nielsen. 2018
81 citations recorded by the source corpus.
Bayesian inference plays an important role in advancing machine learning, but faces computational challenges when applied to complex models such as deep neural networks. Variational inference circumvents these challenges by formulating Bayesian inference as an optimization problem and solving it using gradient-based optimization. In this paper, we argue in favor of natural-gradient approaches which, unlike their gradient-based counterparts, can improve convergence by exploiting the information geometry of the solutions. We show how to derive fast yet simple natural-gradient updates by using a duality associated with exponential-family distributions. An attractive feature of these methods is that, by using natural-gradients, they are able to extract accurate local approximations for individual model components. We summarize recent results for Bayesian deep learning showing the superiority of natural-gradient approaches over their gradient counterparts.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Trevor Darrell
- Marcus Rohrbach
- Sayna Ebrahimi
- Mohamed Elhoseiny
- Mohit Bansal
- Mohammad Emtiyaz Khan
- Richard E. Turner
- Sergey Levine
Citations in the corpus
- Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, methodology): “For Bayesian methods, we compare against VCL and its variations named as VCL with Variational Adam (VCL-Vadam), VCL with Adam and Gaussian natural gradients (VCL-GNG).”
- Practical Deep Learning with Bayesian Principles (2019, background): “The natural-gradient update takes a simple form when estimating exponential-family approximations [23, 22].”
- Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users (2020, methodology): “Natural gradient descent has also been advocated as an optimization tool for variational inference for BNN [65], as the distributions from the exponential family lead to, if parametrized in natural space, a very simple formulation of the natural gradient [39].”
- SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient (2018, methodology): “Moreover, a natural-gradient method is preferable to gradient-based methods when optimizing the parameters of a distribution [3, 15, 18].”