Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs

By Kazuki Osawa, Yohei Tsuji, Yuichiro Ueno, Akira Naruse, Rio Yokota, Satoshi Matsuoka. 2018

39 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “The importance of understanding the benefits of modern parallel hardware has motivated a lot of recent work on training neural networks with larger batch sizes [Goyal et al., 2017, Osawa et al., 2018, McCandlish et al., 2018, Shallue et al., 2018].”
Lookahead Optimizer: k steps forward, 1 step back (2019, methodology): “The fastest convergence we are aware of uses an approximate second-order method to train a ResNet-50 to 75% top-1 accuracy in 35 epochs with 1,024 GPUs [31].”
Practical Deep Learning with Bayesian Principles (2019, background): “[42] for best practice on these hyperparameter values.”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “The use of large-batch methods and reduced floating-point precision has sparked a “race” toward accelerating the training process of specific model/dataset combinations in image processing [84, 123, 176, 191, 256] and language modeling [192, 203, 257].”
Augment Your Batch: Improving Generalization Through Instance Repetition (2020, methodology): “The key to supporting large batch training often involves fine-tuning the base Learning Rate (LR), per-layer LR [41], LR schedules [8, 41], or the optimization step [15, 10, 25].”
Ellipsoidal Trust Region Methods and the Marginal Value of Hessian Information for Neural Network Training (2019, methodology): “[61] Kazuki Osawa, Yohei Tsuji, Yuichiro Ueno, Akira Naruse, Rio Yokota, and Satoshi Matsuoka.”
Augment your batch: better training with larger batches (2019, methodology): “Increasing batch size while mitigating accuracy degradation is actively researched in the ML and systems communities (Goyal et al., 2017; Kurth et al., 2017; Jia et al., 2018; Mikami et al., 2018; Osawa et al., 2018; Ying et al., 2018).”
Limitations of the Empirical Fisher Approximation (2019, background): “There are also several works building upon KFAC which substitute the EF for the Fisher [George et al., 2018, Osawa et al., 2018].”
Adaptive norms for deep learning with regularized Newton methods (2019, background): “The latter observations suggests that advances in distributed second-order algorithms (e.g., Osawa et al. (2018); D¨unner et al. (2018)) constitute a promising direction of research towards the goal of a more widespread use of Newton-type methods in deep learning.”