New Insights and Perspectives on the Natural Gradient Method

By James Martens. 2014

832 citations recorded by the source corpus.

Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of approximate 2nd-order optimization method, where the Fisher information matrix can be viewed as an approximation of the Hessian. This perspective turns out to have significant implications for how to design a practical and robust version of the method. Additionally, we make the following contributions to the understanding of natural gradient and 2nd-order methods: a thorough analysis of the convergence speed of stochastic natural gradient descent (and more general stochastic 2nd-order methods) as applied to convex quadratics, a critical examination of the oft-used "empirical" approximation of the Fisher matrix, and an analysis of the (approximate) parameterization invariance property possessed by natural gradient methods, which we show still holds for certain choices of the curvature matrix other than the Fisher, but notably not the Hessian.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “(Note that this can be seen as a special case of the convergence result derived for convex quadratics in Martens [2014].) Remarkably, each dimension converges exponentially to a steady state risk.”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, methodology): “…value θ∗i , controlled by the λ hyperparameter, as follows: L(θ) = LFT (θ) + ∑ i λ 2 Fi,i(θi − θ∗i )2 (1) For efficient training, we use the empirical Fisher (Martens, 2014): diagonal elements are approximated through squaring first-order gradients from a sample of instances, recomputed…”
Training Neural Networks with Fixed Sparse Masks (2021, methodology): “It can be shown [34, 37] that as δ → 0 , to second order, where F θ ∈ R | θ |×| θ | is the Fisher information matrix [14, 4], deﬁned as Given this relation, it can be seen that the Fisher information matrix is closely connected to how much each parameter affects the model’s predictions.”
Optimization Methods for Large-Scale Machine Learning (2016, background): “It should also be noted that research on the design of methods inspired by the natural gradient is ongoing and may lead to markedly different algorithms [31, 73, 98].”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “Furthermore, the Fisher information matrix can be seen to define a convex quadratic approximation to the objective function of a probabilistic neural model (Pascanu & Bengio, 2014; Martens, 2014).”
Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “To address the ﬁrst issue, it is common to approximate the Hessian with the Fisher information matrix (FIM; Martens, 2014; Huszár, 2017; Ritter et al., 2018): The FIM is PSD, which ensures that Computing F k may still be impractical if there are many model parameters, and it is therefore common to…”
Posterior Meta-Replay for Continual Learning (2021, methodology): “3/4 in Martens [48]): Note that so far we have considered x ﬁxed, but the model should perform well with respect to the input distribution p ( x ) : where the approximation in the last line comes from an MC estimate using N samples drawn from the joint p ( y | W , x ) p ( x ) .”
Generalized Variational Continual Learning (2020, background): “If we note that H T ≈ N T F T (Martens, 2020), where N T is the number of samples in the T th dataset and F T is the Fisher information matrix, we recover Online EWC with λ = 1 when N 1 = N 2 = ... = N T (with γ = 1 ).”
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks (2019, methodology): “We use the FIM, restricted to the samples we wish to retain, to compute the optimal noise to destroy information, so that a cohort can be forgotten while maintaining good accuracy for the remaining samples.”
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting (2018, methodology): “In order to numerically guarantee this in practice, we use the Fisher Information as an approximation [24] that is positive semi-deﬁnite by construction.”
Where is the Information in a Deep Neural Network? (2019, methodology): “For its relevant properties see [13].”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, methodology): “In practice, for computational efﬁciency the FIM is typically approximated by the empirical FIM to avoid additional backward passes [70].”
Generative replay with feedback connections as a general strategy for continual learning (2018, methodology): “Following the deﬁnitions and notation in [32], the i th diagonal element of F ( k ) is deﬁned as: the previous task, with the strength of each parameter’s penalty depending on how important that parameter is thought to be for the tasks learned so far.”
Note on the quadratic penalties in elastic weight consolidation (2017, methodology): “…near-perfect predictions on task A , we can approximate H as where N A is the number of i. i. d. observations in D A , F ( θ ∗ A ) is the empirical Fisher information matrix on task A [see e. g. section 11 of Martens, 2014] and H prior is the Hessian of the negative log prior − log p ( θ ).”
Lookahead Optimizer: k steps forward, 1 step back (2019, background): “This is even more important in the stochastic setting [25, 43].”
Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (2019, methodology): “We also tried computing the diagonal of ¯F on held-out data, as 5 github.com/thompsonb/sockeye_ewc there is some evidence that estimating Fisher on held out data reduces overfitting in natural gradient descent (Pascanu and Bengio, 2013).”
YellowFin and the Art of Momentum Tuning (2017, methodology): “Empirical Fisher is typically used in practical natural gradient methods (Martens, 2014; Roux et al., 2008; Duchi et al., 2011).”
Three Mechanisms of Weight Decay Regularization (2018, background): “Natural gradient descent, as a variant of second-order methods (Martens, 2014), is able to make more progress per iteration by taking into account the curvature information.”