Natural Gradient Revisited

18 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

The Numerics of GANs (2017, background): “We hence believe that further progress can be made by revisiting second order optimization methods [2, 18] in the context of saddle point problems.”
Representation Learning: A Review and New Perspectives (2012, methodology): “, with good scaling properties) have also been proposed [127], [157].”
Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging (2014, methodology): “We do not have a very good explanation why our parallel training method only works when using Natural Gradient, except to say that the statements in [13] that NG prevents large parameter steps and is more robust to reorderings of the training set, may be relevant.”
Training Neural Networks with Stochastic Hessian-Free Optimization (2013, background): “Recently, [13] provided a relationship between HF, Krylov subspace descent and natural gradient due to their use of the Gauss-Newton curvature matrix.”
Natural Gradient Deep Q-learning (2018, methodology): “Since symmetric KL divergence behaves like a distance measure in inﬁnitesimal form, a Riemannian metric is derived as the Hessian of the divergence of symmetric KL divergence [Pascanu and Bengio, 2013].”