Fixing Weight Decay Regularization in Adam

By Ilya Loshchilov, Frank Hutter. 2017

2,716 citations recorded by the source corpus.

We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor. We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence. Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. Our source code will become available after the review process.

Subfields: Computational Mechanics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Optimization & Training

Citations in the corpus

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, methodology): “All models were optimized with the Adam optimizer (Kingma and Ba, 2015) with weight decay ﬁx (Loshchilov and Hutter, 2017).”
Do Better ImageNet Models Transfer Better? (2018, methodology): “Low dimensional embedding using t-SNE [13] on features from the penultimate layer of Inception v4, for 10 classes from the test set.”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “We use an AdamW optimizer (Loshchilov and Hutter, 2017) with ( β 1 , β 2 ) set to (0 .”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “We use the AdamW (Loshchilov and Hutter, 2019) optimizer, with beta values of 0.9 and 0.95 respectively, and an epsilon of 1.0E−8.”
Galactica: A Large Language Model for Science (2022, methodology): “We train using AdamW with β1 = 0.9, β2 = 0.95 and weight decay of 0.1 (Loshchilov and Hutter, 2017).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, positive): “The additional optimizer currently available is the Adam optimizer (Kingma and Ba, 2014) with an additional weight decay ﬁx, also known as ‘AdamW‘ (Loshchilov and Hutter, 2017).”
On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “…(cid:15) at the common default value of 10 − 8 for Adam and 10 − 10 for RMSProp (Tieleman and Hinton, 2012; Kingma and Ba, 2015; Dozat, 2016; Balles and Hennig, 2017; Loshchilov and Hutter, 2017; Zou and Shen, 2018; Ma and Yarats, 2018; Bernstein et al., 2018; Chen et al., 2019; Zou et al., 2019).”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “We train with the adamw optimizer [37] and a batch size of 1024, distributed over 16 GPUs when using ViT-S/16.”
Training data-efficient image transformers & distillation through attention (2020, methodology): “The best results use the AdamW optimizer with the same learning rates as ViT [15] but with a much smaller weight decay, as the weight decay reported in the paper hurts the convergence in our setting.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “We use weight decay which penalizes the L2 norm of the model parameters [30, 46].”
Scaling Laws for Autoregressive Generative Modeling (2020, methodology): “For math alone we used a weight decay [LH17] 1 Summary of scaling laws—”
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, background): “Michel and Neubig (2018) finetuned the biases of the output softmax in an NMT systems, to personalize the output vocabulary.”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “These benchmarks either use Adam (Kingma and Ba, 2014), AdamW (Loshchilov and Hutter, 2018), or Momentum (Qian, 1999).”
SOCRATES: Introducing Depth in Visual Wildlife Monitoring Using Stereo Vision (2022, methodology): “We used the AdamW optimizer [47] with a global learning rate of 5N10−5 with a batch size of N and a weight decay of 0.”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, methodology): “Our models are trained using AdamW optimizer [30] with a linear warmup and linear decay learning rate scheduler.”
On the Variance of the Adaptive Learning Rate and Beyond (2019, methodology): “Following Loshchilov & Hutter (2018), we decouple weight decays in the vanilla Adam, Adam with warmup and RAdam in our experiments.”
How to train your MAML (2018, methodology): “Annealing the learning rate, either by using step-functions (He et al., 2016) or cosine functions (Loshchilov & Hutter, 2017) has proved vital in learning models with higher generalization power.”