The Natural Gradient by Analogy to Signal Whitening, and Recipes and Tricks for its Use
By Jascha Narain Sohl-Dickstein. 2012
13 citations recorded by the source corpus.
The natural gradient allows for more efficient gradient descent by removing dependencies and biases inherent in a function's parameterization. Several papers present the topic thoroughly and precisely. It remains a very difficult idea to get your head around however. The intent of this note is to provide simple intuition for the natural gradient and its use. We review how an ill conditioned parameter space can undermine learning, introduce the natural gradient by analogy to the more widely understood concept of signal whitening, and present tricks and specific prescriptions for applying the natural gradient to learning problems.
Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Quoc V. Le
- Razvan Pascanu
- Ruslan Salakhutdinov
- Pieter Abbeel
- Mohit Bansal
- Dacheng Tao
In Computational Mechanics
- Surya Ganguli
- Razvan Pascanu
- Ben Poole
- Karen Simonyan
- Koray Kavukcuoglu
- Guillaume Desjardins
- Jascha Narain Sohl-Dickstein
- James Martens
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
- Andrea Vedaldi
In Optimization & Training
- Surya Ganguli
- Razvan Pascanu
- Karen Simonyan
- Ben Poole
- Koray Kavukcuoglu
- Jascha Narain Sohl-Dickstein
- Guillaume Desjardins
- Yoshua Bengio
Citations in the corpus
- Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “The natu-ral gradient can be simply combined with other optimization methods by performing a change of variables, such that in the new parameter space the natural gradient and the ordinary gradient are identical (Sohl-Dickstein, 2012).”
- Natural Neural Networks (2015, methodology): “A similar algorithm to PRONG was later found in [23], where it appeared simply as a thought experiment, but with no amortization or recourse for efficiently computing F .”