Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons

By Shun-ichi Amari, Hyeyoung Park, Kenji Fukumizu. 2000

219 citations recorded by the source corpus.

The natural gradient learning method is known to have ideal performances for on-line training of multilayer perceptrons. It avoids plateaus, which give rise to slow convergence of the backpropagation method. It is Fisher efficient, whereas the conventional method is not. However, for implementing the method, it is necessary to calculate the Fisher information matrix and its inverse, which is practically very difficult. This article proposes an adaptive method of directly obtaining the inverse of the Fisher information matrix. It generalizes the adaptive Gauss-Newton algorithms and provides a solid theoretical justification of them. Simulations show that the proposed adaptive method works very well for realizing natural gradient learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Signal Processing

Citations in the corpus

No more pesky learning rates (2012, methodology): “Approaches based on the natural gradient (Amari et al., 2000) precondition the updates by the empirical Fisher information matrix (estimated by the gradient covariance matrix, or its diagonal approximation), in the simplest case: η i = η 0 /v i ; the “Natural Newton” al-gorithm (Le Roux &…”
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, methodology): “Like Quasi-Newton methods, several methods adaptively adjust learning rates based on local curvature [2, 40], while ADAGRAD [11] applies different learning rates to different dimensions.”
Measuring and regularizing networks in function space (2018, methodology): “That is, we wish to ﬁnd a Fisher-efﬁcient algorithm (see Amari et al. (2000)).”
Large Scale Online Learning (2003, methodology): “The resulting algorithm slightly differs from the Adaptive Natural Gradient algorithm (Amari, Park, and Fukumizu, 1998).”
Adaptive natural gradient learning algorithms for various stochastic models (2000, methodology): “To solve these problems, Amari, Park and Fukumizu (1998) proposed an adaptive method”
Probabilistic Line Searches for Stochastic Optimization (2015, background): “Remark: There is no well-deﬁned probabilistic belief over f (cid:48)(cid:48) and higher derivatives, as sample paths of the Wiener process are almost surely non-differentiable almost everywhere (Adler, 1981, § 2.2).”
Topmoumoute Online Natural Gradient Algorithm (2007, background): “TONGA was designed to address these issues, which it does this by maintaining a low rank approximation of the covariance and by casting both problems of finding the low rank approximation and of computing the natural gradient in a lower dimensional space, thereby attaining a much lower complexity.”
Riemannian metrics for neural networks I: feedforward networks (2013, methodology): “In this context, we also clarify another method found in the literature (Amari et al., 2000; Le Roux et al., 2007).”
Local minima and plateaus in hierarchical structures of multilayer perceptrons (2000, unknown): “In the case of tanh activation function, Chen et al. (1993) showed that any analytic map T : QH ! QH such that f Hx; TuH f Hx; uH is a composition of hidden unit weight interchanges and hidden unit weight sign flips, which later are defined by vj; ~ wj 7! 2vj;2 ~ wj: K.”
Adaptive Sampling for SGD by Exploiting Side Information (2016, background): “Other work on setting the learning rate based on the local curvature and gradients include (Amari et al., 2000; Schaul et al., 2012; Roux & Fitzgibbon, 2010).”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “We note that early variants of this method have been considered previously [10, 25], but we believe we are the first to consider its accuracy, efficiency, and implementability in the context of large-scale deep models, as well as to investigate its extensions.”
Parametric Return Density Estimation for Reinforcement Learning (2010, background): “Because of this, the ordinary gradient is generally different from the steepest direction on the manifold and the optimization process often becomes unstable or falls into a stagnant state, called a plateau (Amari et al., 2000; Kakade, 2002).”
Generalized TD Learning (2011, unknown): “Following Amari et al. (2000), we additionally simplify update equation (27) as Q̂−1t = (1+ εt)Q̂ −1 t−1− εtQ̂−1t−1ŵ∗t−1∂θε(zt , θ̂t−1)⊤Q̂−1t−1, (28) Algorithm 2 Optimal TD Learning Initialize α̂0, β̂0, θ̂0, Q̂−10 = εIm, a1, a2 {ε andIm denote a small constant and anm×m identical matrix.} for t =…”
Universal statistics of Fisher information in deep neural networks: mean field approach (2018, methodology): “This is known as the empirical FIM and often appears in practice [27–31]:”
A Stochastic Quasi-Newton Method for Online Convex Optimization (2007, unknown): “The natural gradient (NG) algorithm (Amari et al., 2000) incorporates the Riemannian metric tensor ∇ ] into the stochastic gradient update: with gains η t typically set by (7), and ˆ G t an estimate of The Sherman-Morrison formula can be employed to directly update ˆ G − 1 t , reducing the…”
Deep learning via Hessian-free optimization (2010, methodology): “Moreover, for standard neural nets it can also be performed without the need to evaluate non-linear functions.”
Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent (2011, methodology): “Since H is unknown in advance, methods for adaptively estimating H is proposed (Bottou and LeCun, 2005; Amari et al., 2000).”