Scaling SGD Batch Size to 32K for ImageNet Training

By Yang You, Igor Gitman, Boris Ginsburg. 2017

424 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2019, methodology): “Adaptive optimization methods [23], [6], [24], [25] are crucial to achieving SOTA performance and accuracy for effective model training of large models.”
Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Increasing interest has focused on large batch training (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), in an attempt to increase the step size and reduce the number of parameter updates required to train a model.”
Group Normalization (2018, background): “We suspect that the hierarchical batching property may be an essential factor underlying the recently prevalent large-batch distributed training (Goyal et al. 2017; Gitman and Ginsburg 2017).”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “Optimization We use the LARS optimizer [57] with a cosine decay learning rate schedule [58], without restarts, over 1000 epochs, with a warm-up period of 10 epochs.”
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, background): “On the other hand, (Krizhevsky, 2014; Goyal et al., 2017; You et al., 2017; Smith et al., 2017) observed that rescaling the step size η ∼ m works well in practice when m is not too large.”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, background): “, 32K) to train models orders of magnitude more quickly than previously possible [16, 46].”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, methodology): “The most successful in this line of research is the L ARS algorithm (You et al., 2017), which was initially proposed for training R ES N ET .”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “Such a solution is equivalent to single-threaded SGD with large mini-batch sizes (Goyal et al., 2017; You et al., 2017a,b, 2019).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Considering other optimizers, Table 5 below shows that post-local SGD can improve upon LARS 6 (You et al., 2017a).”
Local SGD Converges Fast and Communicates Little (2018, background): “Recent work in [11, 43] explores various limitations of this approach, as in general it is reported that performance degrades for too large mini-batch sizes [14, 19, 42].”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Regarding the second problem, this study (You et al., 2017) suggested an algorithm based on layer-wise adaptive rate scaling (LARS).”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, background): “However, recent literature on large-scale supervised learning of image classifiers [44–46] explored much larger batch sizes and showed that strong scaling was possible by carefully tuning learning rate and initialization of the neural network.”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “Optimization is then performed against this aggregate loss using the LARS optimizer [118].”