Kronecker-factored Curvature Approximations for Recurrent Neural Networks

By James Martens, Jimmy Ba, Matt Johnson. 2018

106 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “For recurrent neural networks, we use the approximation presented in Section 3.4 of Martens et al. (2018).”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, unknown): “Martens & Grosse (2015) and follow-up works (Ba et al., 2017; George et al., 2018; Martens et al., 2018) proposed to approximate the Hessian based on the structure of neural networks.”
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information (2021, methodology): “Singh and Alistarh [37] showed that it can be scaled to DNNs by blockwise approximation, and that it provides better approximations of the loss than K-FAC-based [44], or diagonal [39] approximations [7], leading to more accurate pruning.”
Extreme Tensoring for Low-Memory Preconditioning (2019, methodology): “[MG15, GKS18, MBJ18] have investigated tensor-factorized preconditioners.”
BackPACK: Packing more into backprop (2019, background): “This work sparked a new line of research to improve the second-order approximation (Grosse & Martens, 2016; Botev et al., 2017; Martens et al., 2018; George et al., 2018).”
Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks (2019, background): “Recently, scalable approximations to natural gradient descent have shown practical success in a variety of tasks and architectures [Martens and Grosse, 2015, Grosse and Martens, 2016, Wu et al., 2017, Zhang et al., 2018a, Martens et al., 2018].”
Merging Models with Fisher-Weighted Averaging (2021, methodology): “Some work on natural gradient descent may prove relevant for model merging such as using Kronecker-factorized Fisher matrices as an alternative to the diagonal approximation employed in this paper [37, 18, 38].”
WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, methodology): “When one proceeds to the case of convolutional or recurrent neural networks, the Kronecker structure needs to be specially designed by making further approximations [40, 41].”
A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (2020, unknown): “Then, KFAC is extended to convolutional neural networks (Grosse and Martens, 2016), recurrent neural networks (Martens et al., 2018) and variational Bayesian neural networks (Bae et al., 2018; Zhang et al., 2018) and shows signiﬁcant speedups during training.”
The Case for Full-Matrix Adaptive Regularization (2018, methodology): “We ﬁnd recent deep learning applications of second-order methods (e.g. [MG15, MBJ18]) to be intriguing, though outside the scope of this paper.”