Distributed Second-Order Optimization using Kronecker-Factored Approximations
By Jimmy Ba, Roger Baker Grosse, James Martens. 2016
126 citations recorded by the source corpus.
Subfields: Artificial Intelligence, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Ruslan Salakhutdinov
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Sergey Levine
In Optimization & Training
- Roger Baker Grosse
- Dan Alistarh
- Yoshua Bengio
- Guodong Zhang
- David Kristjanson Duvenaud
- James Martens
- Jimmy Ba
- Jascha Narain Sohl-Dickstein
Citations in the corpus
- Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Other optimizers, such as KFAC (Martens and Grosse, 2015; Grosse and Martens, 2016; Ba et al., 2017), or optimization techniques designed”
- Large scale distributed neural network training through online distillation (2018, methodology): “, 2017) as the number of workers, and thus the effective batch size, increases and we might hope that algorithms like KFAC (Ba et al., 2017) would make better use of large batches.”
- Continual Learning With Extended Kronecker-Factored Approximate Curvature (2020, methodology): “[2] develops a version of K-FAC that distributes the computation of gradients and additional quantities required by K-FAC across multiple machines.”
- Three Mechanisms of Weight Decay Regularization (2018, methodology): “It has been shown that K-FAC scales very favorably to larger mini-batches compared to SGD, enjoying a nearly linear relationship between mini-batch size and per-iteration progress for medium-to-large sized mini-batches (Martens & Grosse, 2015; Ba et al., 2016).”
- Demystifying Parallel and Distributed Deep Learning (2018, methodology): “9,150,243,254] Training Distribution Model Consolidation Ensemble Learning[114,148,217] Knowledge Distillation[10,96] Model Averaging: Direct[20,33,168,266], Elastic[121,155,248,258], Natural Gradient[11,195] Optimization Algorithms First-Order[43,123,126,141,145,163,227] Second-Order[11,28,56,94,13”
- Noisy Natural Gradient as Variational Inference (2017, methodology): “In training, we adopted learning rate selection strategy adopted by Ba et al. (2016).”
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, unknown): “Martens & Grosse (2015) and follow-up works (Ba et al., 2017; George et al., 2018; Martens et al., 2018) proposed to approximate the Hessian based on the structure of neural networks.”
- Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks (2018, methodology): “The previous studies that used K-FAC to train ResNet50 on ImageNet [5] also were not considering large minibatches and were only training with mini-batch size of 512 on 8 GPUs.”
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis (2018, methodology): “Factored approximations such as KFAC (Martens & Grosse, 2015; Ba et al., 2017) approximate each block as a Kronecker product of two much smaller matrices, both of which can be estimated and inverted more efficiently than the full block matrix, since the inverse of a Kronecker product of two matrices…”
- WoodFisher: Efficient second-order approximations for model compression (2020, methodology): “As a result, K-FAC has found several applications in the last few years in: optimization (Ba et al., 2016; Osawa et al., 2019), pruning (Zeng and Urtasun, 2019; Wang et al.”
- A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (2020, unknown): “, 2018; Bae et al., 2018) and Eigenvaluecorrected Kronecker Factorization (EKFAC) (George et al., 2018). KFAC was first proposed in Martens and Grosse (2015) for FNNs, which provides a successful approximate natural gradient optimizer.”
- Kronecker-factored Curvature Approximations for Recurrent Neural Networks (2018, methodology): “Kronecker-factor Approximate Curvature (Martens & Grosse, 2015) (K-FAC) is a 2nd-order optimization method which has been shown to give state-of-the-art performance on large-scale neural network optimization tasks (Ba et al., 2017).”
- Non-Parametric Calibration for Classification (2019, background): “…which strive to improve uncertainty estimation only for neural networks, including Bayesian neural networks (MacKay, 1992, Gal, 2016) and Laplace approximations (LA) (Martens and Grosse, 2015, Ba et al., 2017), our aim is a framework that is not based on tuning a specific classification method.”
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis (2019, methodology): “2016)’s “doubly fac-tored” Kronecker approximation for layers whose input feature maps are too large, we ignore the correlation among the spatial locations within the same input channel.”
- WoodFisher: Efficient Second-Order Approximation for Neural Network Compression (2020, background): “As a result, K-FAC has found several applications in the last few years in: optimization [38, 39], pruning [21, 22], reinforcement-learning [51], etc.”