On the Variance of the Adaptive Learning Rate and Beyond

By Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han. 2019

2,272 citations recorded by the source corpus.

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Adversarial Training for Large Neural Language Models (2020, methodology): “…on unlabeled text has proven very effective to improve generalization performance for a variety of downstream tasks, as exempliﬁed by BERT (Devlin et al., 2018) and other transformer-based models (Liu et al., 2019c; Radford et al., 2018; Clark et al., 2020; Dong et al., 2019; Bao et al., 2020).”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “For panoptic segmentation, we strictly follow Panoptic-DeepLab [20], except using a linear warm up Radam [61] Lookahead [96] optimizer (with the same learning rate 0.”
Learning Visual Representations with Caption Annotations (2020, methodology): “For that, ﬁrst, we train separate VGG16 networks on VG or COCO for 120K iterations using RAdam [39] with batches of size 128, initial learning rate 1 e − 3, weight decay 1 e − 5, and the learning rate is decayed by 0 .”
On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “Zaheer et al. (2018) and Liu et al. (2019) consider (cid:15) values orders of magnitude larger than the standard default.”
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence (2020, methodology): “, 2020)(4) (iii) rectified Adam (RADAM) (Liu et al., 2019) (iv) Look-ahead optimizer (Zhang et al.”
Getting to 99% Accuracy in Interactive Segmentation (2020, methodology): “We train our model with the RAdam optimiser [40] with a learning rate of 10 − 5 , with a single image in each batch.”
On Layer Normalization in the Transformer Architecture (2020, methodology): “Liu et al. (2019a) suggests that the warm-up stage plays a role in reducing the undesirably signiﬁcant variance in Adam in the early stage of model training.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Specifically, we train our models with 32 TPU cores for 100k (400k for main results) iterations (54 epochs), a batch size of 64, Radam [62] Lookahead [104], a ‘poly’ schedule learning rate of 10 (3 ⇥ 10 for MaX-DeepLab-L), a backbone learning rate multiplier of 0.”
Albumentations: fast and flexible image augmentations (2018, methodology): “Training ran for 100 epochs and RAdam optimizer [69] was used with the starting learning rate of 10−3 and cosine annealing to 10−6.”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, methodology): “We train all generative models with the RAdam optimizer (Liu et al., 2020) with a learning rate of 1 e − 4 for tabular data and 3 e − 4 for MNIST.”
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting (2020, methodology): “We used the RAdam (Liu et al., 2019) optimizer with a learning rate of 0 .”
iTAML: An Incremental Task-Agnostic Meta-learning Approach (2020, methodology): “For training, we use RAdam [16] with initial learning rate of 0 .”
Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram (2019, methodology): “Models were trained for 400 K steps with RAdam optimizer ( (cid:15) = 1 e − 6 ) to stabilize training [27].”
On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (2020, background): “Another line of work investigates optimization difﬁculties when pre-training transformer-based language models (Xiong et al., 2020; Liu et al., 2020).”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, methodology): “In the future, we would like to investigate the relationship between POSTNORM and PRENORM when using other optimizers such as RADAM [34], which has been shown to improve Transformer training without warmup.”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing (2020, background): “We focused especially on the complex interplay between learning rate and the number of warm-up steps which was shown to be crucial to prevent deeper layers of creating instability during training [53] and speed-up model convergence [54].”