Large Batch Training of Convolutional Networks
By Yang You, Igor Gitman, Boris Ginsburg. 2017
967 citations recorded by the source corpus.
A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
- Dacheng Tao
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Optimization & Training
- Peter Richtárik
- Tengyu Ma
- Martin Jaggi
- Yuanzhi Li
- Quoc V. Le
- Sebastian U. Stich
- Jason D. Lee
- Zeyuan Allen-Zhu
In Scaling & Learning Theory
- Yuanzhi Li
- Jason D. Lee
- Tengyu Ma
- Colin Wei
- Zeyuan Allen-Zhu
- Daniel Soudry
- Michael W. Mahoney
- Simon Shaolei Du
Citations in the corpus
- A Simple Framework for Contrastive Learning of Visual Representations (2020, positive): “To stabilize the training, we use the LARS optimizer (You et al., 2017) for all batch sizes.”
- Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “Nevertheless, batching is not a panacea: it puts pressure on the limited HBM memory capacity, and very large batch sizes can slow model convergence rates (Shallue et al., 2018; You et al., 2017; Lanchantin et al., 2020; Anil et al., 2021).”
- Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Increasing interest has focused on large batch training (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), in an attempt to increase the step size and reduce the number of parameter updates required to train a model.”
- Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “We use the LARS optimizer [32] (with a momentum of 0.”
- Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, background): “Note that LARS [28] offers layer-wise adaptive learning rate and is reported to be effective for extremely large batch sizes (beyond 16K).”
- An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “It uses the LARS optimizer [46] with a 4096 batch size [9], lr=0.”
- Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “We use a weight decay of 10 − 5 , LARS optimizer [55] and O1 mixed-precision optimization from Apex library.”
- Exploring Simple Siamese Representation Learning (2020, methodology): “We also note that the standard SGD optimizer does not work well when the batch is too large (even in supervised learning [14, 38]), and our result is lower with a 4096 batch.”
- ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “The Adam optimizer is applied instead of the LARS because of its effective capability to train models with small batch sizes [48].”
- High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “In particular, You et al. (2017) propose LARS, a momentum variant which sets the norm of the parameter update to be a fixed ratio of the parameter norm, completely ignoring the gradient magnitude.”
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, positive): “, LARS [62], cosine learning rate [38, 42] and the MLP projection head [10].”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “Further, algorithmic innovations such as momentum [30, 4], adaptivity [17, 41, 42], and clipping [38, 39, 43] are critical for success in deep learning applications and need to be incorporated into the client updates, replacing the SGD update of FEDAVG.”
- Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Considering other optimizers, Table 5 below shows that post-local SGD can improve upon LARS 6 (You et al., 2017a).”
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Regarding the second problem, this study (You et al., 2017) suggested an algorithm”