Natural gradient descent for on-line learning

By Magnus Rattray, D. Saad, Shun-ichi Amari. 1998

Natural gradient descent is an on-line variable-metric optimization algorithm which utilizes an underlying Riemannian parameter space. We analyze the dynamics of natural gradient descent beyond the asymptotic regime by employing an exact statistical mechanics description of learning in two-layer feed-forward neural networks. For a realizable learning scenario we find significant improvements over standard gradient descent for both the transient and asymptotic stages of learning, with a slower power law increase in learning time as task complexity grows.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, background): “ld not get stalled near such plateaus. In Park et al. (2000) such plateaus are found near singularities of the functional manifold, providing a nice framework to study them (as is done for example in Rattray et al. (1998) where they hypothesize that such singularities behave like repellors for the d”
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (2014, background): “The slow learning dynamics within this submanifold originates from saddle point structures (caused by permutation symmetries among hidden units), and their associated plateaus (Rattray et al. , 1998; Inoue et al. , 2003).”
Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition (2015, background): “In general, symmetry is known to generate saddle points, and variants ofgradient descent often perform reasonably in these cases (see (Saad and Solla, 1995), (Rattray et al., 1998), (Inoue et al., 2003)).”
Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons (2000, background): “The statistical physical approach elucidated an ideal behavior of natural gradient learning (Rattray et al., 1998), avoiding plateaus.”
An Online Learning Approach to Model Predictive Control (2019, methodology): “This rule uses the natural Riemannian metric of distributions to normalize the effects of different parameterizations of the same distribution [25].”
On the saddle point problem for non-convex optimization (2014, background): “It is argued in Rattray et al. (1998); Inoue et al. (2003) that natural gradient descent can address certain saddle point structures effectively.”
On Natural Learning and Pruning in Multilayered Perceptrons (2000, methodology): “A kind of standard problem used for analyzing neural network learning in the statistical mechanics framework (see, e.g., Rattray et al., 1998).”
Stochastic Cubic Regularization for Fast Nonconvex Optimization (2017, methodology): “Empirically, second-order methods are known to perform better than ﬁrst-order methods on a variety of nonconvex problems [Rattray et al., 1998, Martens, 2010, Regier et al., 2017].”
Modelling the influence of data structure on learning in neural networks (2019, methodology): “The use of second-order gradient descent methods such natural gradient descent [38] can shorten the plateau [39], but we would like to focus on the more practically relevant case of ﬁrst-order SGD. Yoshida et al. [17] recently showed that length and existence of the plateau depend on the…”
Natural conjugate gradient training of multilayer perceptrons (2006, background): “As shown by Amari and his coworkers [1,13], this can be put to advantage when on– line MLP training is considered.”