Super-convergence: very fast training of neural networks using large learning rates

By Leslie N. Smith, Nicholay Topin. 2018

1,797 citations recorded by the source corpus.

In this paper, we describe a phenomenon, which we named “super-convergence”, where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate. A insight that allows super-convergence training is that large learning rates regularize the training, hence requiring a reduction of all other forms of regularization in order to preserve an optimal regularization balance. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. Experiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet datasets, and resnet, wide-resnet, densenet, and inception architectures. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. The architectures and code to replicate the figures in this paper are available at github.com/lnsmith54/super-convergence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “Because our goal in this work is to construct a static dataset, rather than an interactive environment, we are willing to sacriﬁce rendering speed to achieve larger scale and greater photorealism.”
The Two Regimes of Deep Network Training (2020, background): “Soon after, Smith and Nicholay (2017) concludes that a single period of that pattern is sufﬁcient to obtain good performance and the schedule is named 1-cycle .”
TransMorph: Transformer for unsupervised medical image registration (2021, neutral): “18 (Darken and Moody, 1991; Looney, 1996; Zeiler et al., 2013; Smith and Topin, 2019).”
Gaitgraph: Graph Convolutional Network for Skeleton-Based Gait Recognition (2021, methodology): “Adam optimizer is used with a 1-cycle learning rate [22] and a weight decay penalty of 1e-5.”
Chemformer: a pre-trained transformer for computational chemistry (2021, positive): “The one-cycle learning rate schedule [45] was used for fine-tuning, for both sequence-to-sequence and discriminative tasks.”
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning (2020, methodology): “We refrain from using more elaborate optimization techniques including works on super-convergence (Smith & Topin, 2019) and stochastic weight averaging (SWA) (Izmailov et al., 2018).”
Choose a Transformer: Fourier or Galerkin (2021, methodology): “All attention-based models match the parameter quota of the baseline, and are trained using the loss in (2) with the same 1cycle scheduler [78] for 100 epochs.”
GNOT: A General Neural Operator Transformer for Operator Learning (2023, methodology): “We train all models with AdamW (Loshchilov & Hutter, 2017) optimizer with the cycle learning rate strategy (Smith & Topin, 2019) or the exponential decaying strategy.”
A Generalized Supervised Contrastive Learning Framework (2022, methodology): “of architectures on ImageNet [8], which mainly consists of RandAugment [7], Mixup [46] and CutMix [45], one-cycle learning rate scheduling [35], exponential-moving average of model weights [22] and knowledge distillation (KD [21]).”
Mish: A Self Regularized Non-Monotonic Neural Activation Function (2019, methodology): “Figure 22 also demonstrates the comparison analysis between Mish and Swish while using One Cycle Policy [21,22].”
Rethinking the Hyperparameters for Fine-tuning (2020, neutral): “There are many literatures on hyperparameter selection for training neural networks from scratch, mostly on batch size, learning rate and weight decay (Goyal et al., 2017; Smith et al., 2018; Smith & Topin, 2019).”
MOMENT: A Family of Open Time-series Foundation Models (2024, background): “In all experiments that follow, unless mentioned otherwise, we fine-tune MOMENT-Large with a batch size of 64, and one cycle learning rate schedule with a peak learning rate between 5 e − 5 and 1 e − 3 (Smith & Topin, 2019).”
DaViT: Dual Attention Vision Transformers (2022, methodology): “We use a simple triangular learning rate schedule [45] as in [51].”
Better Diffusion Models Further Improve Adversarial Training (2023, methodology): “We use the cyclic learning rate schedule with cosine annealing (Smith & Topin, 2019), where the initial learning rate is set to 0.2.”