Cyclical Learning Rates for Training Neural Networks

By Leslie N. Smith. 2015

2,966 citations recorded by the source corpus.

It is known that the learning rate is the most important hyper-parameter to tune for training deep neural networks. This paper describes a new method for setting the learning rate, named cyclical learning rates, which practically eliminates the need to experimentally find the best values and schedule for the global learning rates. Instead of monotonically decreasing the learning rate, this method lets the learning rate cyclically vary between reasonable boundary values. Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. This paper also describes a simple way to estimate "reasonable bounds" – linearly increasing the learning rate of the network for a few epochs. In addition, cyclical learning rates are demonstrated on the CIFAR-10 and CIFAR-100 datasets with ResNets, Stochastic Depth networks, and DenseNets, and the ImageNet dataset with the AlexNet and GoogLeNet architectures. These are practical tools for everyone who trains neural networks.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “STLR modifies triangular learning rates (Smith, 2017) with a short increase and a long decay period, which we found key for good performance.”
Fine-tuned Language Models for Text Classification (2018, methodology): “w much smaller the lowest LR is from the maximum LR max, and tis the learning rate at iteration t. We generally use cutfrac= 0:1, ratio= 32and max= 0:01. STLR modiﬁes triangular learning rates (Smith, 2017) with a short increase and a long decay period, which we found key for good performance.5 I”
Fixing Weight Decay Regularization in Adam (2017, background): “Adjusting (e.g., decreasing) η ( i ) min and η ( i ) max at every i -th restart (see also Smith (2016)) could potentially improve performance, but we do not consider that option in our experiments because it would involve additional hyperparameters.”
Decentralized gradient methods: does topology matter? (2020, neutral): “A constant learning rate has been set using the conﬁguration rule from (Smith, 2017) described in Appendix G. Interest-ingly, for a given ML problem, when the dataset is split randomly, this procedure has led to choose the learning rate independently of the average node degree.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “The first problem is typically solved with automated HPO or the cyclical LR method (Smith, 2017).”
Neural Prototype Trees for Interpretable Fine-grained Image Recognition (2020, methodology): “We evaluate on CUB-200-2011 [49] with 200 bird species (CUB) and Stanford Cars [29] with 196 car types (CARS), since both were used by ProtoPNet [8].”
Population Based Training of Neural Networks (2017, background): “PBT also allows for adaptive hyperparameters during training, which has been shown, especially for learning rates, to improve optimisation (Loshchilov & Hutter, 2016; Smith, 2017; Massé & Ollivier, 2015).”
Three Factors Influencing Minima in SGD (2017, background): “It has been observed that a cyclic learning rate (CLR) schedule leads to better generalization Smith (2015).”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “, 2018) and the efficacy of alternative learning rate schedules (Smith, 2017; Smith & Topin, 2018; Li & Arora, 2020).”
A Baseline for Few-Shot Image Classification (2019, neutral): “learning rates [47] and half-precision distributed training on 8 Nvidia V100 GPUs [48, 49] to reduce training time.”
XNAS: Neural Architecture Search with Expert Advice (2019, background): “[26, 38], as the later additionally suggests a way for obtaining an empirical upper bound on the learning rate.”
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks (2019, background): “Additionally, there are several alternative learning rate schedules proposed for SGD, such as warm-restarts [28] and [33].”
Fast is better than free: Revisiting adversarial training (2020, background): “Cyclic learning rate Introduced by Smith (2017) for improving convergence and reducing the amount of tuning required when training networks, a cyclic schedule for a learning rate can drastically reduce the number of epochs required for training deep networks (Smith & Topin, 2018).”
An Exponential Learning Rate Schedule for Deep Learning (2019, background): “There are other recently discovered exotic LR schedules, e.g. Triangular LR schedule(Smith, 2017) and Cosine LR schedule(Loshchilov & Hutter, 2016), and our exponential LR schedule is an extreme example of LR schedules that become possible in presence of BN.”
Exploring loss function topology with cyclical learning rates (2017, background): “The learning rate (LR) range test and cyclical learning rates (CLR) are described in (Smith, 2015) and (Smith, 2017).”
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020, methodology): “Following Merkx et al. (2019) we trained the model using the Adam optimizer (Kingma and Ba, 2015) with a cyclical learning rate schedule (Smith, 2017).”
Comparing Rewinding and Fine-tuning in Neural Network Pruning (2020, methodology): “inds only the learning rate, continuing to train the weights from their values at the end of training (like ﬁne-tuning). This is similar to the learning rate schedule used by cyclical learning rates (Smith, 2017). In this paper, we compare ﬁne-tuning, weight rewinding, and learning rate rewinding as”