New perspectives on the natural gradient method

By James Martens. 2014

78 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Measuring and regularizing networks in function space (2018, methodology): “As explained in Martens (2014) and in Pascanu & Bengio (2013), the proper method is to calculate G from the predictive (output) distribution of the network, P θ ( y | x ) .”
Note on the quadratic penalties in elastic weight consolidation (2017, methodology): “…near-perfect predictions on task A , we can approximate H as where N A is the number of i. i. d. observations in D A , F ( θ ∗ A ) is the empirical Fisher information matrix on task A [see e. g. section 11 of Martens, 2014] and H prior is the Hessian of the negative log prior − log p ( θ ).”
Stochastic natural gradient descent draws posterior samples in function space (2018, background): “As the learning rate → 0 , the change in function ( f ω t +1 ( X ) − f ω t ( X )) during an update becomes independent of the model parameterization (Martens, 2014).”
A Distributional View on Multi-Objective Policy Optimization (2020, unknown): “In general, invariance to reparameterization of the function approximator has been investigated in optimization literature resulting in, for example, natural gradient methods (Martens, 2014).”
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise (2019, background): “5(b), we give the error bar of the maximum eigenvalue of the Hessian over different runs, which indicates the modification gives a roughly consistent estimate.”
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks (2019, background): “It is also closely connected to Gauss-Newton optimization, suggesting it should achieve fast convergence in certain settings [Pascanu and Bengio, 2013, Martens, 2014, Botev et al., 2017].”
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, methodology): “Since the pretrained model convergences on the source tasks, H(θ∗) can be approximated with the empirical Fisher information matrix F (θ∗) (Martens, 2014):”