Revisiting Natural Gradient for Deep Networks

By Razvan Pascanu, Yoshua Bengio. 2013

448 citations recorded by the source corpus.

We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models. The contributions of this paper are as follows. We show the connection between natural gradient and three other recently proposed methods for training deep models: Hessian-Free (Martens, 2010), Krylov Subspace Descent (Vinyals and Povey, 2012) and TONGA (Le Roux et al., 2008). We describe how one can use unlabeled data to improve the generalization error obtained by natural gradient and empirically evaluate the robustness of the algorithm to the ordering of the training set compared to stochastic gradient descent. Finally we extend natural gradient to incorporate second order information alongside the manifold information and provide a benchmark of the new algorithm using a truncated Newton approach for inverting the metric matrix instead of using a diagonal approximation of it.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Adam: A Method for Stochastic Optimization (2014, methodology): “…a preconditioner that adapts to the geometry of the data, since (cid:98) v t is an approximation to the diagonal of the Fisher information matrix (Pascanu & Bengio, 2013); however, Adam’s preconditioner (like AdaGrad’s) is more conservative in its adaption than vanilla NGD by preconditioning…”
On the Number of Linear Regions of Deep Neural Networks (2014, methodology): “The MLPs were trained on the same synthetic dataset using a conjugate natural gradient (Pascanu and Bengio 2014) which was used to minimize the effect of optimization.”
Trust Region Policy Optimization (2015, methodology): “For additional exposition on the use of Hessian-vector products for optimizing neural network objectives, see (Martens & Sutskever, 2012) and (Pascanu & Bengio, 2013).”
Overcoming catastrophic forgetting in neural networks (2016, background): “F has three key properties (20): (i) It is equivalent to the second derivative of the loss near a minimum, (ii) it can be computed from first-order derivatives alone and is thus easy to calculate even for large models, and (iii) it is guaranteed to be positive semidefinite.”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “The recently introduced Hessian-Free Optimization technique (Martens, 2010) and the Krylov Subspace Descent methods (Vinyals & Povey, 2012) use an extended Gauss-Newton approximation of the Hessian, therefore they can be interpreted as versions of natural gradient descent (Pascanu & Bengio, 2014).”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “There has been substantial work in explicitly incorporating second-order information through the Hessian matrix in Newton methods (Liu and Nocedal, 1989; Nocedal and Wright, 2006; Martens, 2010) or through the Fisher information matrix in natural gradient methods (Amari, 1998; Roux et al., 2008; Pas”
Training Neural Networks with Fixed Sparse Masks (2021, background): “It can be shown [34, 37] that as δ → 0 , to second order, where F θ ∈ R | θ |×| θ | is the Fisher information matrix [14, 4], deﬁned as Given this relation, it can be seen that the Fisher information matrix is closely connected to how much each parameter affects the model’s predictions.”
PGQ: Combining policy gradient and Q-learning (2016, background): “Natural policy gradient can also be interpreted as putting a constraint on the KL-divergence at each step of the policy improvement (Amari, 1998; Kakade, 2001; Pascanu & Bengio, 2013).”
Representation Learning: A Review and New Perspectives (2012, methodology): “Natural gradient (Amari, 1998) methods that can be applied to networks with millions of parameters (i.e. with good scaling properties) have also been proposed (Le Roux et al., 2008b; Pascanu and Bengio, 2013).”
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (2014, methodology): “Similar to (Pascanu and Bengio, 2014), we use Lagrange multipliers to obtain the solution of this constrained optimization.”
How to Construct Deep Recurrent Neural Networks (2013, methodology): “We ﬁnd some of the recently introduced approaches, such as advanced regularization methods (Pascanu et al. , 2013a) and advanced optimization algorithms (see, e.g., Pascanu and Bengio, 2013; Martens, 2010), to be promising candidates.”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, background): “For some good recent discussion and analysis of the natural gradient, see Arnold et al. (2011); Martens (2014); Pascanu and Bengio (2014).”
Sharp Minima Can Generalize For Deep Nets (2017, background): “See Amari (1998); Pascanu & Bengio (2014) for more details.”
Neural Optimizer Search with Reinforcement Learning (2017, neutral): “…this difﬁculty (e.g., Riedmiller & Braun (1992); LeCun et al. (1998); Schraudolph (2002); Martens (2010); Le et al. (2011); Duchi et al. (2011); Zeiler (2012); Martens & Sutskever (2012); Schaul et al. (2013); Pascanu & Bengio (2013); Pascanu et al. (2013); Kingma & Ba (2015); Ba et al. (2017)).”
Overcoming Multi-Model Forgetting (2019, positive): “This approximation falsely presupposes that the parameters ( θ 1 , θ s ) are independent, but it has already proven effective (Kirkpatrick et al., 2017; Pascanu & Bengio, 2014).”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “Furthermore, the Fisher information matrix can be seen to deﬁne a convex quadratic approximation to the objective function of a probabilistic neural model (Pascanu & Bengio, 2014; Martens, 2014).”
Overcoming Catastrophic Forgetting by Incremental Moment Matching (2017, methodology): “For covariance, we use the inverse of a Fisher information matrix, following [8, 22].”
DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “Trace-restricted K-FAC (TK-FAC) [10] is a trace-exact variant which corrects the norm of the factorized FIM by further tracking the trace of FIM.”
Posterior Meta-Replay for Continual Learning (2021, background): “Our discussion on this topic is inspired by the derivation of the natural gradient [59], that allows to cast parameter updates into distribution updates.”
Supermasks in Superposition (2020, neutral): “EWC uses the Fisher Information matrix [36].”