Natural Gradient Works Efficiently in Learning

By Shun-ichi Amari. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Layer Normalization (2016, methodology): “…following: where, f ( · ) is the transfer function that is the analog of the non-linearity in neural networks, f (cid:48) ( · ) is the derivative of the transfer function, η ( · ) is a real valued function and c ( · ) is the log partition function. φ is a constant that scales the output variance.”
Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities (2013, methodology): “It is well known that second-order optimization methods such as the natural gradient [1] or Newton’s method decrease the number of required iterations compared to the basic gradient descent, but they cannot be easily used with high-dimensional models due to heavy computations with large matrices.”
Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, background): “Finally, the natural gradient (Amari, 1998) can greatly accelerate optimization by removing the effect of dependencies and relative scalings between parameters.”
Revisiting Natural Gradient for Deep Networks (2013, background): “Natural gradient can be traced back to Amari’s work on information geometry (Amari, 1985) and its application to various neural networks (Amari et al. , 1992; Amari, 1997), though a more in depth introduction can be found in Amari (1998); Park et al. (2000); Arnold et al. (2011).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “Unit natural gradients or “Unitwise Fisher’s scoring” (Kurita, 1993) are based on natural gradients for perceptrons (Amari, 1998; Yang & Amari, 1998).”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “There has been substantial work in explicitly incorporating second-order information through the Hessian matrix in Newton methods (Liu and Nocedal, 1989; Nocedal and Wright, 2006; Martens, 2010) or through the Fisher information matrix in natural gradient methods (Amari, 1998; Roux et al., 2008; Pas”
The Numerics of GANs (2017, background): “We hence believe that further progress can be made by revisiting second order optimization methods [2, 18] in the context of saddle point problems.”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “It is often more eﬀective to deﬁne an appropriate Riemannian metric that respects the loss surface (Amari, 2012), which allows much steeper descent directions to be identiﬁed within a local neighborhood ( e.g., Amari (1998); Martens & Grosse (2015)).”
PGQ: Combining policy gradient and Q-learning (2016, background): “Natural policy gradient can also be interpreted as putting a constraint on the KL-divergence at each step of the policy improvement (Amari, 1998; Kakade, 2001; Pascanu & Bengio, 2013).”
Progressive Neural Networks (2016, neutral): “Fig.”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “By contrast, the natural gradient method [3, 4] aims to be invariant with respect to all differentiable and invertible transformations.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, neutral): “The well-known natural gradient (Amari, 1998) is deﬁned as F − 1 ∇ h ( θ ) .”
Sharp Minima Can Generalize For Deep Nets (2017, methodology): “Note that a α-scale transformation will not affect the generalization, as the behavior of the function is identical.”
Variational Inference: A Review for Statisticians (2016, background): “In gradient-based optimization, the natural gradient accounts for the geometric structure of probability parameters (Amari 1982, 1998).”
Supervised Dimensionality Reduction via Distance Correlation Maximization (2016, methodology): “of the natural gradient [Amari, 1998, Nishimori and Akaho, 2005]. Our contribution in this paper is as follows: We propose a new formulation for supervised dimensionality reduction that is based on a dependency criterion called Distance Correlation, [Szekely et al., 2007]. This setup is free of dist”