Adaptive natural gradient learning algorithms for various stochastic models

By Hyeyoung Park, Shun-ichi Amari, Kenji Fukumizu. 2000

180 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Signal Processing

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, background): “Natural gradient can be traced back to Amari’s work on information geometry (Amari, 1985) and its application to various neural networks (Amari et al. , 1992; Amari, 1997), though a more in depth introduction can be found in Amari (1998); Park et al. (2000); Arnold et al. (2011).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “The recently introduced Hessian-Free Optimization technique (Martens, 2010) and the Krylov Subspace Descent methods (Vinyals & Povey, 2012) use an extended Gauss-Newton approximation of the Hessian, therefore they can be interpreted as versions of natural gradient descent (Pascanu & Bengio, 2014).”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “se the discovery of the natural gradient algorithm is closely associated with information geometry, nearly all its applications involve density estimation [3,31] or conditional probability estimation [117,73,97] using objective functions that are closely connected to the KL divergence. For instance,”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “The traditional and more intuitive approach of using an exponentially decayed running average (e.g. LeCun et al., 1998; Park et al., 2000) works better, at least pre-asymptotically, as it is able to naturally “forget” very old contributions to the estimate (which are based on stale parameter values)”
Universal statistics of Fisher information in deep neural networks: mean field approach (2018, methodology): “, softmax outputs), these differences are basically limited to the multiplication of activations in the output layer [30].”
Diagonal Rescaling For Neural Networks (2017, methodology): “Many authors have proposed second order stochastic gradient algorithms for neural networks [Becker and LeCun, 1989, Park et al., 2000, Ollivier, 2013, Martens and Grosse, 2015, Marceau-Caron and Ollivier, 2016].”
Exact natural gradient in deep linear networks and its application to the nonlinear case (2018, background): “By definition, the natural gradient is the product of the inverse of the P ×P Fisher information matrix F with the P -dimensional gradient vector, where P is the number of network parameters (often in the millions) [Yang and Amari, 1998; Amari et al., 2000; Park et al., 2000].”
A Stochastic Quasi-Newton Method for Large-Scale Optimization (2014, methodology): “al bounds. The method does, however, require knowledge of the underlying distribution of the training points (x;z), and the Fisher information matrix must be inverted. These concerns are addressed in [18], which presents an adaptive method for computing the inverse Fisher Information matrix in the c”
Limitations of the Empirical Fisher Approximation (2019, methodology): “The generalized Gauss-Newton [Schraudolph, 2002] and natural gradient descent [Amari, 1998] methods have inspired a line of work on approximate second-order optimization [Martens, 2010, Botev et al., 2017, Park et al., 2000, Pascanu and Bengio, 2014, Ollivier, 2015].”
Dynamics of Learning Near Singularities in Layered Networks (2008, methodology): “The natural gradient method (Amari, 1998; Park, Amari, & Fukumizu, 2000 ) gives one solution to it. It is interesting to investigate it further in the present context (see Cousseau et al., in press).”
Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons (2000, methodology): “This is a preliminary study; more general performances of the proposed method will be reported in the future by Park, Amari, and Fukumizu (2000).”