Don't Decay the Learning Rate, Increase the Batch Size

By Samuel L. Smith, Pieter-Jan Kindermans, Quoc V. Le. 2017

1,135 citations recorded by the source corpus.

It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times. We can further reduce the number of parameter updates by increasing the learning rate $\epsilon$ and scaling the batch size $B \propto \epsilon$. Finally, one can increase the momentum coefficient $m$ and scale $B \propto 1/(1-m)$, although this tends to slightly reduce the test accuracy. Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning. We train ResNet-50 on ImageNet to $76.1\%$ validation accuracy in under 30 minutes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “, better loss as a function of tokens seen) earlier in training, while larger batch sizes are beneficial later in training due to better gradient estimates (Smith et al., 2018; McCandlish et al., 2018), and (2) larger batch sizes result in larger matrix multiplication dimensions, which increases TPU”
An Empirical Model of Large-Batch Training (2018, methodology): “Adaptive batch sizes may also have other beneﬁts, such as replacing adaptive learning rates [SKYL17] and managing the proportion of gradient noise during training.”
Compressive Transformers for Long-Range Sequence Modelling (2019, background): “Reducing the optimisation frequency increases the effective batch size, which has also been shown to be prefer-8 able to learning rate decay in image modelling (Smith et al., 2018).”
Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, neutral): “Similar empirical results have been reported for neural networks [25].”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “…ET - 50 training on ImageNet in a few minutes (You et al., 2018; Iandola et al., 2016; Codreanu et al., 2017; Akiba et al., 2017; Jia et al., 2018; Smith et al., 2017; Martens & Grosse, 2015; Devarakonda et al., 2017; Mikami et al., 2018; Osawa et al., 2018; You et al., 2019; Yamazaki et al.,…”
Towards Federated Learning at Scale: System Design (2019, background): “While much successful work on deep learning has used asynchronous training, e.g., Dean et al. (2012), recently there has been a consistent trend towards synchronous large batch training, even in the data center (Goyal et al., 2017; Smith et al., 2018).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “, H > 1) after performing the first learning rate decay2, which is the phase corresponding to the turnover point of the model that gradually and eventually converges to the minima (Robbins & Monro, 1985; Smith et al., 2017).”
Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Properties (2020, background): “[26] Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, neutral): “Prior work has observed that increasing the batch size tends to slightly reduce the test accuracy, and have proposed various methods to reduce the drop in accuracy [4, 2, 5].”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, background): “Practical SGD implementations generally decay the learning rate or increase the mini-batch size (Smith et al., 2017a; Goyal et al., 2017), in order to reduce the variance of the gradient updates.”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “ization (LeCun et al., 2012; Keskar et al., 2017; Goyal et al., 2017; Smith & Le, 2018; Smith et al., 2018) and the efficacy of alternative learning rate schedules (Smith, 2017; Smith & Topin, 2018; Li & Arora, 2020).”
Understanding Batch Normalization (2018, background): “This has indeed been observed in practice in the context of parallelizing neural networks [14, 49] and derived in other theoretical models [24].”