An Exponential Learning Rate Schedule for Deep Learning

By Zhiyuan Li, Sanjeev Arora. 2019

272 citations recorded by the source corpus.

Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The following new results are shown about BN with weight decay and momentum (in other words, the typical use case which was not considered in earlier theoretical analyses of stand-alone BN. 1. Training can be done using SGD with momentum and an exponentially increasing learning rate schedule, i.e., learning rate increases by some $(1 +\alpha)$ factor in every epoch for some $\alpha >0$. (Precise statement in the paper.) To the best of our knowledge this is the first time such a rate schedule has been successfully used, let alone for highly successful architectures. As expected, such training rapidly blows up network weights, but the net stays well-behaved due to normalization. 2. Mathematical explanation of the success of the above rate schedule: a rigorous proof that it is equivalent to the standard setting of BN + SGD + StandardRate Tuning + Weight Decay + Momentum. This equivalence holds for other normalization layers as well, Group Normalization, LayerNormalization, Instance Norm, etc. 3. A worked-out toy example illustrating the above linkage of hyper-parameters. Using either weight decay or BN alone reaches global minimum, but convergence fails when both are used.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Big Transfer (BiT): General Visual Representation Learning (2019, background): “This counter-intuitive behavior stems from the interaction of weight decay and normalization layers [29,32].”
Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “During training, we reduce the learning rate once the accuracy has plateaued [58, 59].”
Understanding the Role of Training Regimes in Continual Learning (2020, methodology): “Secondly, when combined with Batch Normalization [28], L 2 regularization leads to an Exponential Learning Rate Schedule [45], meaning that in each iteration, multiplying the initial learning rate by (1 + α ) for some α > 0 that depends on the momentum and weight decay rate.”
Rethinking Experience Replay: a Bag of Tricks for Continual Learning (2020, background): “Exponentialbased rules for decaying the learning rate were early introduced in literature to speed up the learning process [32], [33].”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “…related to the scale and distribution of SGD noise, e.g., the relationship between batch size, learning rate, and generalization (Keskar et al., 2017; Smith & Le, 2018; Smith et al., 2018) and the efﬁcacy of alternative learning rate schedules (Smith, 2017; Smith & Topin, 2018; Li & Arora, 2019).”
Regular Polytope Networks (2021, background): “As shown in [16], fixed classifiers are also related to BatchNorm [15] and learning rate schedules.”
Class-incremental Learning with Pre-allocated Fixed Classifiers (2020, background): “It is shown recently in [51] that CNNs with a fixed classifier and batch-norm in each layer establish a principle of equivalence between different learning rate schedules.”
Large Scale Learning of General Visual Representations for Transfer (2019, background): “This counter-intuitive behavior stems from the interaction of weight decay and normalization layers [26,28].”
Online Continual Learning For Interactive Instruction Following Agents (2024, methodology): “We utilize the ExponentalLR (Li & Arora, 2019) and ResetLR (Loshchilov & Hutter, 2016) schedulers with γ = 0 .”
The Two Regimes of Deep Network Training (2020, background): “Li and Arora (2019) even showed that, counter-intuitively, an exponentially increasing schedule could also be effective.”
Bad Global Minima Exist and SGD Can Reach Them (2019, background): “Before we proceed, we would like to note that one can potentially force vanilla SGD to escape bad initializers (or any initializers for that matter) by employing non-standard, adaptive learning schedules such as those in [27, 28], or by using “unnatural”, or enormously large learning rate for a few ”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “At the heart of our approach are the expected magnitudes of weights, activations, gradients, and weight updates, all of which have been identified as important factors in previous work (e.g., [1, 3, 6, 7, 9, 37, 39– 41, 59, 73, 74]).”
Evolving Normalization-Activation Layers (2020, background): “…is aligned with some previous ﬁndings that scale invariance might play a useful role from the optimization point of view (Hoffer et al., 2018; Li & Arora, 2020). γ + β The numerator of EvoNorm-S0 resembles the Swish activation function (Ramachandran et al., 2018) and its denominator…”
On Learning Rates and Schrödinger Operators (2020, methodology): “Empirically, deep learning models work well with non-decaying schedules such as cyclical learning rates [LH16, Smi17] (see also the review [Sun19]), with recent theoretical justiﬁcation [LA19].”
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (2020, background): “Arora et al. [45] argued that batch normalization reduces the importance of tuning the learning rate, while Li and Arora [46] showed that models trained using batch normalization can converge even if the learning rate increases exponentially during training.”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “Since batch normalization (Ioffe & Szegedy, 2015) is known to have unusual optimization properties (Li & Arora, 2019), it is natural to wonder whether our findings still hold with batch normalization.”
FixNorm: Dissecting Weight Decay for Training Deep Neural Networks (2021, background): “Beyond ELR hypothesis, Li & Arora (2019) derives a closed-form between learning rate, weight decay and momentum, and proposes an exponentially increasing learning rate schedule.”
Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, background): “During training, we reduce the learning rate once the accuracy has plateaued [60, 61].”