SGDR: Stochastic Gradient Descent with Warm Restarts
By Ilya Loshchilov, Frank Hutter. 2016
10,851 citations recorded by the source corpus.
Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL
Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Luke S. Zettlemoyer
- Dacheng Tao
In Computational Mechanics
- Francis R. Bach
- Peter Richtárik
- Zeyuan Allen-Zhu
- Julien Mairal
- Simon Lacoste-Julien
- Jitendra Malik
- Nicolas Loizou
- Michael I. Jordan
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Optimization & Training
- Quoc V. Le
- Yoshua Bengio
- Frank Hutter
- Soham De
- Simon Lacoste-Julien
- Samuel L. Smith
- Martin Jaggi
- Peter Richtárik
Citations in the corpus
- Universal Language Model Fine-tuning for Text Classification (2018, methodology): “5 In Section 5, we compare against aggressive cosine annealing, a similar schedule that has recently been used to achieve state-of-the-art performance in CV (Loshchilov and Hutter, 2017).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “We use the Adam optimizer (Kingma & Ba, 2014) with decoupled weight decay regularization (Loshchilov & Hutter, 2017) applied to all weights that are not gains or biases, and decay the learning rate using a cosine schedule (Loshchilov & Hutter, 2016).”
- A Simple Framework for Contrastive Learning of Visual Representations (2020, positive): “Furthermore, we use linear warmup for the first 10 epochs, and decay the learning rate with the cosine decay schedule without restarts (Loshchilov & Hutter, 2016).”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “…improves training efficiency by modifying loss functions (Clark et al., 2020), model structures/sparsities (Louizos et al., 2018; Gong et al., 2019; Tan & Le, 2019), backpropagation storage requirements (Gruslys et al., 2016), or learning rate schedules (Loshchilov & Hutter, 2017; Li et al., 2020).”
- Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “The learning rate is linearly warmed up from 10 − 7 to 1 for 16K steps and then annealed using a cosine learning rate schedule with a single phase to 0.0001 (Loshchilov and Hutter, 2016).”
- Measuring the Effects of Data Parallelism on Neural Network Training (2018, neutral): “We experimented with the following decay functions: • Constant: d(t) = 1 • Linear: d(t) = 1− (1− α) t T • Cosine (Loshchilov and Hutter, 2017): d(t) = α+ (1−α) 2 ( 1 + cosπ t T )”
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “(2017) and cyclical schedulers based on warm restarts (Loshchilov and Hutter, 2016).”
- Fine-tuned Language Models for Text Classification (2018, methodology): “Fine-tuning with cosine annealing We have observed the best results by fine-tuning with an aggressive cosine annealing schedule (Loshchilov and Hutter, 2017).”
- Gaussian Error Linear Units (GELUs) (2016, methodology): “We train for 50 epochs with the learning rate schedule described in (Loshchilov & Hutter, 2016) ( T 0 = 50 , η = 0 .”
- Adaptive Input Representations for Neural Language Modeling (2018, methodology): “The learning rate is linearly warmed up from 10 − 7 to 1 for 16K steps and then annealed using a cosine learning rate schedule with C cycles (Loshchilov & Hutter, 2016).”
- Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Increasingly, researchers favor sharper decay schedules like cosine decay (Loshchilov & Hutter, 2016) or step-function drops (Zagoruyko & Komodakis, 2016).”
- Exploring Self-Attention for Image Recognition (2020, methodology): “We use the cosine learning rate schedule with base learning rate 0.1 [23].”
- Big Transfer (BiT): General Visual Representation Learning (2019, background): “Numerous improvements to deep network training have recently been introduced, e.g. [55,62,26,35,22,1,64,67,54,60].”
- Learning Video Representations from Textual Web Supervision (2020, methodology): “The learning rate schedule begins with 1500 warmup steps for S3D-G or 2000 for R3D-50 (exponentially increasing from .001 to 1.0 for S3D-G and 1.6 for R3D-50), followed by a cosine-decaying [41] schedule for the remaining steps (140K for S3D-G, 120K for R3D-50).”
- VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “We use linear learning rate warmup [22] for the first 10K iterations followed by cosine decay [96] to zero.”