SGDR: Stochastic Gradient Descent with Restarts

By Ilya Loshchilov, Frank Hutter. 2016

481 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “The learning rate is linearly warmed up from 10 − 7 to 1 for 16K steps and then annealed using a cosine learning rate schedule with a single phase to 0.0001 (Loshchilov and Hutter, 2016).”
Gaussian Error Linear Units (GELUs) (2016, methodology): “We train for 50 epochs with the learning rate schedule described in (Loshchilov & Hutter, 2016) ( T 0 = 50 , η = 0 .”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “The learning rate is linearly warmed up from 10 − 7 to 1 for 16K steps and then annealed using a cosine learning rate schedule with C cycles (Loshchilov & Hutter, 2016).”
Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Increasingly, researchers favor sharper decay schedules like cosine decay (Loshchilov & Hutter, 2016) or step-function drops (Zagoruyko & Komodakis, 2016).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “For all models, we use a cosine learning rate decay schedule (Loshchilov and Hutter, 2016) over 410B tokens, taken as an upper bound for the length of training if compute permitted, and warmup for 375M tokens.”
Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, neutral): “[18] propose a cosine annealing strategy.”
SGDR: Stochastic Gradient Descent with Warm Restarts (2016, methodology): “Our initial arXiv report on SGDR (Loshchilov & Hutter, 2016) inspired a follow-up study by Huang et al. (2016a) in which the authors suggest to take M snapshots of the models obtained by SGDR (in their paper referred to as cyclical learning rate schedule and cosine annealing cycles) right before M…”
A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets (2017, background): “An additional mechanism could be to perform warm restarts (Loshchilov and Hutter, 2017), which was shown to substantially improve any-time performance over reductions of the learning rate at regular intervals.”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, methodology): “Training is warmed up for 500 steps where the learning rate is increased from 1 × 10 −7 to 5 × 10 −3 , and then annealed to 1 × 10 −6 using a cosine schedule (Loshchilov & Hutter, 2016).”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “We optimize them with Adam Kingma & Ba (2015) and a cosine learning rate schedule Loshchilov & Hutter (2016) annealed over 40K update steps for both WSJ and the clean Librispeech training datasets.”
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (2016, methodology): “For CIFAR-10 and CIFAR-100, we train a 40-4 wide residual network (Zagoruyko & Komodakis, 2016) for 50 epochs with stochastic gradient descent using restarts (Loshchilov & Hutter, 2016), the GELU nonlinearity, and standard mirroring and cropping data augmentation.”
Progressive Neural Architecture Search (2017, methodology): “…1, since there are only 136 unique cells with 1 block), we use a maximum cell depth of B = 5 blocks, we use F = 24 ﬁlters in the ﬁrst convolutional cell, we unroll the cells for N = 2 times, and each child network is trained for 20 epochs using initial learning rate of 0.01 with cosine decay [22].”
Born Again Neural Networks (2018, methodology): “(3) Finally, similarly to ensembling multiple snapshots (Huang et al., 2017) of SGD with restart (Loshchilov & Hutter, 2016), we produce Born-Again Network Ensembles (BANE) by averaging the prediction of multiple generations of BANs. f̂k(x) = k∑ i=1 f(x, θi)/k. (4) We find the improvements of the…”
Shake-Shake regularization (2017, methodology): “Training starts with a learning rate of 0.2 and is annealed using a Cosine function without restart (see Loshchilov & Hutter (2016)).”
Pre-trained language model representations for language generation (2019, background): “The learning rate is linearly warmed up from 10 to 1 for 16K steps and then annealed us-ing a cosine learning rate schedule with a single phase to 0.0001 (Loshchilov and Hutter, 2016).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Furthermore, some work like [49, 50] leverage cyclic learning rate schedules either improving the anytime performance of deep neural network training, or ensembling multiple neural networks at no additional training cost.”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “Recent empirical results with non-monotonic learning rate heuristics (Loshchilov & Hutter, 2017) suggest that there are still many unknowns in training neural networks and that many ideas in non-convex optimization can be used to improve it.”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, background): “In particular: • We show that SGD with cyclical [e.g., Loshchilov and Hutter, 2017] and constant learning rates traverses regions of weight space corresponding to high-performing networks.”