Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates

By Leslie N. Smith, Nicholay Topin. 2017

524 citations recorded by the source corpus.

In this paper, we describe a phenomenon, which we named "super-convergence", where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate. A primary insight that allows super-convergence training is that large learning rates regularize the training, hence requiring a reduction of all other forms of regularization in order to preserve an optimal regularization balance. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. Experiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet datasets, and resnet, wide-resnet, densenet, and inception architectures. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. The architectures and code to replicate the figures in this paper are available at this http URL. See this http URL for an application of super-convergence to win the DAWNBench challenge (see this https URL).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Fine-tuned Language Models for Text Classification (2018, neutral): “This is similar to the approach used in (Smith and Topin, 2017).”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “…related to the scale and distribution of SGD noise, e.g., the relationship between batch size, learning rate, and generalization (Keskar et al., 2017; Smith & Le, 2018; Smith et al., 2018) and the efﬁcacy of alternative learning rate schedules (Smith, 2017; Smith & Topin, 2018; Li & Arora, 2019).”
Understanding Batch Normalization (2018, background): “Another line of work of relevance is [48] and [47], where the relationship between various network parameters, accuracy and convergence speed is investigated, the former article argues for the importance of batch normalization to facilitate a phenomenon dubbed ’super convergence’.”
Editable Neural Networks (2020, methodology): “We formulate the SGD editor with up to k steps and learning rate α as: The standard gradient descent editor can be further augmented with momentum, adaptive learning rates (Duchi et al. (2010); Zeiler (2012)) and other popular deep learning tricks (Kingma & Ba (2014); Smith & Topin (2017)).”
Fast is better than free: Revisiting adversarial training (2020, methodology): “…to a lesser extent, other adversarial training methods) can be drastically accelerated using standard techniques for efficient training of deep networks, including e.g. cyclic learning rates (Smith & Topin, 2018), mixed-precision training (Micikevicius et al., 2017), and other similar techniques.”
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, background): “( iii ) This cyclical stepsize shares the advantage of the “super-convergence” property discussed in Smith & Topin (2017): cSG-MCMC can accelerate convergence for DNNs by up to an order of magnitude.”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “We optimize using the AdamW [21] optimizer with the one-cycle learning rate scheduler [37].”
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation (2022, methodology): “We train for 600 epochs using AdamW [35] and a one-cycle learning rate schedule [49] with a maximal learning rate of 10.”
RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching (2021, methodology): “We use a one-cycle learning rate schedule [30] with a minimum learning rate of 1e−4.”
Snapshot Distillation: Teacher-Student Optimization in One Generation (2018, background): “, using super-convergence [38] to alleviate the drawback of weaker convergence.”
Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds (2022, methodology): “Adam solver with onecycle learning strategy [8] is used for optimization.”
A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay (2018, background): “Section 4.1 provides updated experiments on super-convergence (Smith & Topin, 2017).”
Continual Learning of Semantic Segmentation using Complementary 2D-3D Data Representations (2021, methodology): “SGD is used with a 1cycle learning rate schedule [27] and a batch size of 16 to adapt the parameters.”
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark (2018, methodology): “ai team used cyclic learning rates [43] and 8 NVIDIA V100 GPUs to train a custom Wide ResNet [51] on CIFAR10 to 94% accuracy in less than 3 minutes (cost of $1.”