Three Mechanisms of Weight Decay Regularization

By Guodong Zhang, Chaoqi Wang, Bowen Xu, Roger Baker Grosse. 2018

289 citations recorded by the source corpus.

Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Do Better ImageNet Models Transfer Better? (2018, background): “The improvement in transfer performance when incorporating batch normalization scale parameters may relate to changes in effective learning rates [70, 76].”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “Indeed, recent theoretical works [Jacot et al., 2018, Du et al., 2019, Zhang et al., 2019a] show that for wide enough networks, the weights stay close enough to the initialization for the linearized approximation to remain accurate.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “We use weight decay which penalizes the L2 norm of the model parameters [30, 46].”
Initialization and Regularization of Factorized Neural Layers (2021, methodology): “We support this hypothesis using experiments analogous to Zhang et al. (2019) by comparing training a low-rank ResNet-20 with FD to training it with no decay but at each step normalizing all BatchNormed layers to have the same Frobenius norm as the FD run.”
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring (2019, methodology): “We train our models using Adam (Kingma & Ba, 2015) with a ﬁxed learning rate of 0.002 and weight decay (Zhang et al., 2018) with a ﬁxed value of 0 .”
An Exponential Learning Rate Schedule for Deep Learning (2019, background): “None of the above analyses incorporated weight decay, but (Zhang et al., 2019; Hoffer et al., 2018a; van Laarhoven, 2017; Page; Wu) argued qualitatively that weight decay makes parameters have smaller norms, and thus the effective learning rate, η w w t (cid:107) 2 is larger.”
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence (2019, methodology): “Several authors [38, 15, 35] have observed that WD increases the effective learning rate ⌘eff,t = ⌘t/kwtk(2)2, where ⌘t is the learning rate at epoch t and kwtk(2)2 is the squared-norm of weights at epoch t, by decreasing the weight norm, which increases the effective gradient noise, which promotes ”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “At the heart of our approach are the expected magnitudes of weights, activations, gradients, and weight updates, all of which have been identified as important factors in previous work (e.g., [1, 3, 6, 7, 9, 37, 39– 41, 59, 73, 74]).”
Batch Normalization is a Cause of Adversarial Vulnerability (2019, methodology): “We use “F” to denote Fixup (Zhang et al., 2019b), for noise were: Gaussian— 9 .”
FixNorm: Dissecting Weight Decay for Training Deep Neural Networks (2021, methodology): “Van Laarhoven (2017) first introduces the ELR hypothesis and provides derivations for different optimizers, while both Hoffer et al. (2018); Zhang et al. (2018) give additional evidence supporting the hypothesis.”
Eigenvalue Corrected Noisy Natural Gradient (2018, unknown): “, 2016] and could generalize as well as SGD [Zhang et al., 2018].”
EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis (2019, background): “Depthwise separable convolution has been proven to be effective in designing lightweight models (Howard et al., 2017; Chollet, 2017; Zhang et al., 2018b; Ma et al., 2018).”
An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise (2019, background): “…devoted to this in deep learning; for example, scaling the learning rate or batch-size (Smith 1This corresponds to the matrix-variate Gaussian noise in Zhang et al. (2018) and Le, 2017; Goyal et al., 2017; Hoffer et al., 2017) to augment gradient noise in order to encourage better generalization.”
Rethinking the Hyperparameters for Fine-tuning (2020, unknown): “It also seems contradicting to previous ﬁnding (Zhang et al., 2018) that weight decay functions as increasing ELR by decreasing weight norms.”
A Trace-restricted Kronecker-Factored Approximation to Natural Gradient (2020, unknown): “We mainly follow the experimental setup without weight decay in (Zhang et al. 2019).”