Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient

By Shun-ichi Amari. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, background): “, 2011; Chapelle and Erhan, 2011), Krylov Subspace Descent (Vinyals and Povey, 2012), natural gradient descent (Amari, 1997; Park et al., 2000), TONGA (Le Roux et al.”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, background): “The amount of curvature, and thus the success of our optimization, is not invariant to reparameterization [1]: there may be multiple equivalent ways of parameterizing the same model, some of which are much easier to optimize than others.”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “The recently introduced Hessian-Free Optimization technique (Martens, 2010) and the Krylov Subspace Descent methods (Vinyals & Povey, 2012) use an extended Gauss-Newton approximation of the Hessian, therefore they can be interpreted as versions of natural gradient descent (Pascanu & Bengio, 2014).”
Training Neural Networks with Fixed Sparse Masks (2021, background): “It can be shown [34, 37] that as δ → 0 , to second order, where F θ ∈ R | θ |×| θ | is the Fisher information matrix [14, 4], deﬁned as Given this relation, it can be seen that the Fisher information matrix is closely connected to how much each parameter affects the model’s predictions.”
Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression (2012, background): “Robbins and Monro (1951) and Amari (1997) show that a sequence of learning rates wt = ct −1 is asymptotically efficient in stochastic gradient descent as the number of iterations N goes to infinity, but this conclusion rests on strong assumptions on the functional form of the objective function (e.”
A Bayesian Approach to Online Learning (2006, methodology): “A recent modification of (4.2) has been introduced by Amari [Amari 1996, Amari 1997], who replaces the scalar learning rate η(t) by a tensor.”
Probabilistic independent component analysis for functional magnetic resonance imaging (2004, background): “sequently improved, extended and modified [15] and its relation to maximum-likelihood (ML) estimation and redundancy reduction was investigated [16].”
Merging Models with Fisher-Weighted Averaging (2021, methodology): “The Fisher information matrix F ✓ [16, 3] of a neural network p ✓ ( y | x ) trained to predict an output y from input data x is a | ✓ | ⇥ | ✓ | positive semideﬁnite matrix given by the formula It can be shown that the Fisher information matrix coincides with the Hessian H at modes of the…”
Natural Gradient Works Eciently in Learning (result): “This is an extended version of an earlier NIPS paper (Amari, 1996) including new results.”
Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information (1997, methodology): “From Y = WX, we have H(Y ) = H(X) + log jW j: Applying (21) and the above expressions to (3), we have I(Y ;W) H(X) log jW j+ n2 log(2 e) + n X a=1F ( a3; a4) (22) On the other hand, the following approximation of the marginal entropy (Comon 1994) is obtained by using the Edgeworth expansion of pa(ya”
Blind source separation of more sources than mixtures using overcomplete representations (1999, background): “natural gradient extension [1], [2] which speeds convergence.”
TIES-Merging: Resolving Interference When Merging Models (2023, methodology): “Fisher Merging [45] goes beyond simple averaging to identify the importance of individual parameters using Fisher Information Matrix [18, 3, 34] and uses it to weigh the parameters in each model when merging.”
Natural Gradient Works Efficiently in Learning (1998, methodology): “This is an extended version of an earlier article (Amari, 1996), including new results.”
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks (2019, methodology): “Because training large neural networks is costly, there has been much interest in using second-order optimization to speed up training [Becker and LeCun, 1989, Martens, 2010, Martens and Grosse, 2015], and in particlar natural gradient descent [Amari, 1998, 1997].”
Stochastic Learning (2003, methodology): “The Natural Gradient algorithm [3] provides a principled way to search a Riemannian domain.”
Complexity Issues in Natural Gradient Descent Method for Training Multilayer Perceptrons (1998, methodology): “The following lemma, proved in Yang and Amari (1997a), gives the relations betweenM andM.”