Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix

By Roger Baker Grosse, Ruslan Salakhutdinov. 2015

66 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “FActorized Natural Gradient (FANG) (Grosse & Salakhudinov, 2015) estimates the natural gradient via an approximation of the Fisher information matrix by a Gaussian graphical model.”
Importance Weighted Autoencoders (2015, methodology): “RBMs trained with centering or FANG methods achieve a similar performance of around -100 nats (Grosse & Salakhudinov, 2015).”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “This problem can be addressed through the use of various approximations to the Fisher (e.g Le Roux et al., 2008; Ollivier, 2015; Grosse and Salakhudinov, 2015; Martens and Grosse, 2015) that are designed to be easier to compute, to store and finally to invert, than the exact Fisher.”
Natural Neural Networks (2015, methodology): “By modeling the Fisher matrix as the covariance of a sparsely connected Gaussian graphical model, FANG [19] represents a general formalism for exploiting model structure to efficiently compute the natural gradient.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “Generally speaking, normalizing gradients is similar to secondorder optimization [173], [174], [175], [176] or coordinate-wise adaptive learning rate based methods [7], [177], [178], but with the goal of exploiting the layer-wise structural information in DNNs.”
Data-Dependent Path Normalization in Neural Networks (2015, unknown): “Optimization methods are inherently tied to a choice of geometry over parameter space, which in turns induces a geometry over model space, which plays an important role in regularization and generalization (Neyshabur et al., 2015c).”
Centered Weight Normalization in Accelerating Training of Deep Neural Networks (2017, methodology): “Several studies [12, 11] recently have attempted to use the pre-conditioning techniques to improve the conditioning of the cost curvature.”
Implicit Regularization in Deep Learning (2017, background): “However it is known to be computationally demanding and thus many approximations have been proposed [92, 93, 94].”
Kronecker-factored Curvature Approximations for Recurrent Neural Networks (2018, methodology): “Such an approach to computing a tractable Fisher approximations was first explored by Grosse & Salakhutdinov (2015) for RBMs, although our use of it here is substantially different, and requires additional mathematical machinery.”
A Survey of Optimization Methods From a Machine Learning Perspective (2019, background): “In addition, the factorized natural gradient (FNG) [150]”