Decoupled Weight Decay Regularization

By Ilya Loshchilov, Frank Hutter. 2017

36,546 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, positive): “We ﬁnd that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “Our models are trained using the AdamW optimizer (Loshchilov and Hutter, 2017), with the following hyper-parameters: β1 = 0.”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “For all models, we use the AdamW optimizer (Loshchilov and Hutter, 2017), with β1 = 0.”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “We use the Adam optimizer (Kingma & Ba, 2014) with decoupled weight decay regularization (Loshchilov & Hutter, 2017) applied to all weights that are not gains or biases, and decay the learning rate using a cosine schedule (Loshchilov & Hutter, 2016).”
End-to-End Object Detection with Transformers (2020, methodology): “We train DETR with AdamW [26] setting the initial transformer’s learning rate to 10−4, the backbone’s to 10−5, and weight decay to 10−4.”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “We use AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of 3 e − 4 and weight decay of 0 .”
Technical report on Conversational Question Answering (2019, methodology): “We use AdamW (Loshchilov and Hutter, 2017) as our optimizer with a learning rate of 5e-5 and a batch size of 48.”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “For our optimizer we utilize Adam (Kingma & Ba, 2014) with weight decay (Loshchilov & Hutter, 2019) λ = 0 .”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, background): “999 ) and a decoupled weight decay (Loshchilov and Hutter, 2019) of 0 .”
Unified Scaling Laws for Routed Language Models (2022, methodology): “All were optimized with AdamW [Loshchilov and Hutter, 2018] and ZeRO Stage 1 was used to shard the optimizer state [Rajbhandari et al.”
Improving language models by retrieving from trillions of tokens (2021, methodology): “All models are trained using AdamW (Loshchilov and Hutter, 2019) with a weight decay parameter of 0.1.”
RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, methodology): “We look at several datasets from GLUE, i.e. MRPC Dolan and Brockett [2005], SST-2 Socher et al. [2013], QNLI Rajpurkar et al. [2016], STS-B Al-Natsheh [2017], QQP Chen et al. [2018b] and MNLI Williams et al. [2018].”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “The optimiser was AdamW (Loshchilov and Hutter, 2019) with a 5e-5 learning rate and a 0.01 weight decay.”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “We use the AdamW (Loshchilov and Hutter, 2019) optimizer, with beta values of 0.9 and 0.95 respectively, and an epsilon of 1.0E−8.”