On the adequacy of untuned warmup for adaptive optimization

By Jerry Ma, Denis Yarats. 2019

83 citations recorded by the source corpus.

Adaptive optimization algorithms such as Adam (Kingma and Ba, 2014) are widely used in deep learning. The stability of such algorithms is often improved with a warmup schedule for the learning rate. Motivated by the difficulty of choosing and tuning warmup schedules, recent work proposes automatic variance rectification of Adam's adaptive learning rate, claiming that this rectified approach ("RAdam") surpasses the vanilla Adam algorithm and reduces the need for expensive tuning of Adam with warmup. In this work, we refute this analysis and provide an alternative explanation for the necessity of warmup based on the magnitude of the update term, which is of greater relevance to training stability. We then provide some "rule-of-thumb" warmup schedules, and we demonstrate that simple untuned warmup of Adam performs more-or-less identically to RAdam in typical practical settings. We conclude by suggesting that practitioners stick to linear warmup with Adam, with a sensible default being linear warmup over 2 / (1 - β₂) training iterations.

Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

Are we pretraining it right? Digging deeper into visio-linguistic pretraining (2020, methodology): “Following [34], we set warmup iterations Table 1: Performance on different downstream datasets when we ﬁnetune pretrained models.”
Why are Adaptive Methods Good for Attention Models? (2020, background): “Since then, many works studied different interesting aspects of adaptive methods, see [1, 6, 13, 16–18, 27, 30, 32–36].”
Domain-Independent Dominance of Adaptive Methods (2019, methodology): “We hypothesize that this is result of RAdam incorporating learning rate warmup (see Ma & Yarats [28] for more details), which is not typically employed when training LSTMs, and AdaBound’s adoption of SGD-like dynamics early in training [37].”
Improving Transformer Optimization Through Better Initialization (2020, unknown): “Despite signiﬁcant progress, there have been conﬂicting conclusions on learning rate warmup from previous work (Ma & Yarats, 2019; Liu et al., 2020).”