Topmoumoute Online Natural Gradient Algorithm

By Nicolas Le Roux, Pierre-Antoine Manzagol, Yoshua Bengio. 2007

220 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities (2013, methodology): “dient descent, but they cannot be easily used with high-dimensional models due to heavy computations with large matrices. In practice, it is possible to use a diagonal or block-diagonal approximation [7] of the Fisher information matrix or the Hessian. Gradient descent can be seen as an approximatio”
Revisiting Natural Gradient for Deep Networks (2013, methodology): “While the probabilistic derivation requires the centered covariance, in Le Roux et al. (2008) it is argued that one can use the uncentered covariance U resulting in a simpliﬁed formula which is sometimes confused with the metric derived by Amari: The discrepancy comes from the fact that the eq. is…”
No more pesky learning rates (2012, positive): “Similar ideas are adopted in TONGA (Le Roux et al., 2008).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “tworks the Fisher information matrix is typically too expensive to compute, different approximations of the natural gradient have been proposed. Topmoumoute Online natural Gradient Algorithm (TONGA) (LeRoux et al., 2008) uses a low-rank approximation of natural gradient descent. FActorized Natural G”
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation (2019, methodology): “There has been substantial work in explicitly incorporating second-order information through the Hessian matrix in Newton methods (Liu and Nocedal, 1989; Nocedal and Wright, 2006; Martens, 2010) or through the Fisher information matrix in natural gradient methods (Amari, 1998; Roux et al., 2008; Pas”
Representation Learning: A Review and New Perspectives (2012, background): “, with good scaling properties) have also been proposed [127], [157].”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, positive): “For convenience we will deﬁne the following additional no-Optimizing tation: D v = d L ( y, f ( x, θ )) d v = − d log p ( y | x, θ ) d v and g i = D s i Algorithm 1 (in Appendix A) shows how to compute the gradient D θ of the loss function of a neural network using standard backpropagation.”
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence (2018, positive): “pected loss-gradient covariance matrix. By construction (outer product of gradients), F is positive semi-deﬁnite (PSD) which also makes it highly attractive for second-order optimization techniques [1,19,2,10,17]. Thus, when !0, computing KL-divergence 1 2k k2 F is equivalent to computing distanc”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “This problem can be addressed through the use of various approximations to the Fisher (e.g Le Roux et al., 2008; Ollivier, 2015; Grosse and Salakhudinov, 2015; Martens and Grosse, 2015) that are designed to be easier to compute, to store and finally to invert, than the exact Fisher.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “Diag-onal and low-rank online approximations of natural gradient (Le Roux et al. , 2008; Le Roux et al. , 2011) have been proposed and shown to speed-up training in some contexts.”
Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging (2014, methodology): “In [7], the Fisher matrix was divided into diagonal blocks and each block was approximated by a low-rank matrix.”
A Kronecker-factored approximate Fisher matrix for convolution layers (2016, methodology): “First, one can impose a structure on F enabling tractable inversion; for instance LeCun et al. (1998) approximates it as a diagonal matrix, TONGA (Le Roux et al., 2008) uses a more flexible low-rank-within-block-diagonal structure, and factorized natural gradient (Grosse & Salakhutdinov, 2015)…”
Natural Neural Networks (2015, background): “In one of its implementations, it targets the same block diagonal as PRONG while also exploiting 1As the Fisher and thus ψθ depend on the parameters θ(t), these should be indexed with a time superscript, which we drop for clarity. the low rank structure of these blocks for efficiency, reminiscent of”
Experiments on Parallel Training of Deep Neural Network using Model Averaging (2015, methodology): “Stampede nodes run Linux 2.6.32 with batch services managed by the Simple Linux Utility for Resource Management (SLURM).”
Meta-Curvature (2019, methodology): “In addition, the problems of computational and spatial complexity for training deep networks can be effectively handled thanks to recent approximation techniques [24, 38].”
LOGAN: Latent Optimisation for Generative Adversarial Networks (2019, methodology): “More speciﬁcally, we use the empirical Fisher F (cid:48) with Tikhonov damping, as in TONGA (Roux et al., 2008) F (cid:48) is cheaper to compute compared with the full Fisher, since g is already available.”