L2 Regularization versus Batch and Weight Normalization

By T. van Laarhoven. 2017

344 citations recorded by the source corpus.

Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate. We investigate this dependence, both in theory, and experimentally. We show that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate. This leads to a discussion on other ways to mitigate this issue.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Do Better ImageNet Models Transfer Better? (2018, background): “The improvement in transfer performance when incorporating batch normalization scale parameters may relate to changes in effective learning rates [70, 76].”
Wide neural networks of any depth evolve as linear models under gradient descent (2019, neutral): “It has already been adopted in several recent works (van Laarhoven, 2017; Karras et al., 2018; Jacot et al., 2018; Du et al., 2018; Park et al., 2018).”
Big Transfer (BiT): General Visual Representation Learning (2019, background): “lot, weight decay 1e-5). However, this setting eventually results in an under-performing nal model. This counter-intuitive behavior stems from the interaction of weight decay and normalization layers [26,28]. Low weight decay results in growing weight norms, which in turn results in a diminishing ee”
Fixup Initialization: Residual Learning Without Normalization (2019, methodology): “…surface may be alleviated for residual networks by using smaller initialization than the standard ones such as Xavier or He’s initialization in residual branches. van Laarhoven (2017); Hoffer et al. (2018) studied the effect of (batch) normalization and weight decay on the effective learning rate.”
Regularisation of neural networks by enforcing Lipschitz continuity (2018, methodology): “Interestingly, it has been shown that weight decay (adding an ℓ 2 penalty term to the loss function) provides no regularisation beneﬁt when used in conjunction with these methods (van Laarhoven, 2017), but it does change the eﬀective learning rate of commonly used optimisation algorithms.”
Three Mechanisms of Weight Decay Regularization (2018, background): “But empirically, weight decay appears to signiﬁcantly improve generalization for BN networks (e.g. see Figure 1). van Laarhoven (2017) observed that weight decay, by reducing the norm of the weights, increases the effective learning rate.”
An Exponential Learning Rate Schedule for Deep Learning (2019, background): “None of the above analyses incorporated weight decay, but (Zhang et al., 2019; Hoffer et al., 2018a; van Laarhoven, 2017; Page; Wu) argued qualitatively that weight decay makes parameters have smaller norms, and thus the effective learning rate, η w w t (cid:107) 2 is larger.”
On the Margin Theory of Feedforward Neural Networks (2018, methodology): “Because batchnorm can cause the regularizer to have different effects (van Laarhoven, 2017), we remove batchnorm layers and train a 16 layer deep WideResNet.”
Rethinking the Hyperparameters for Fine-tuning (2020, positive): “Weight Decay and Learning Rate The relationship between weight decay and effective learning rate is recently well-studied (van Laarhoven, 2017; Zhang et al., 2018; Loshchilov & Hutter, 2018).”
Large Scale Learning of General Visual Representations for Transfer (2019, background): “This counter-intuitive behavior stems from the interaction of weight decay and normalization layers [26,28].”
Positional Normalization (2019, methodology): “…methods was initially attributed to “reducing internal covariate shift” in hidden layers [29, 41], an array of recent studies [2, 3, 5, 25, 48, 60, 69, 77] has provided evidence that BN changes the loss surface and prevents divergence even with large step sizes [5], which accelerates training…”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “Another research direction is to analyze the effect of weight decay [189] when combined with scale-invariant normalization methods [92], [142], [161], [185], [190], [191], [192].”
Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs (2018, methodology): “To prevent the scale of weights from becoming too large, we adopt the Normalizing Weights [26] technique.”
Shredder: Learning Noise to Protect Privacy with Partial DNN Inference on the Edge (2019, methodology): “That’s why we employ a formulation opposite to L2 regularization [36], in order to make the magnitude of noise elements greater.”
FixNorm: Dissecting Weight Decay for Training Deep Neural Networks (2021, background): “Van Laarhoven (2017) first introduces the ELR hypothesis and provides derivations for different optimizers, while both Hoffer et al. (2018); Zhang et al. (2018) give additional evidence supporting the hypothesis.”
Shredder: Learning Noise Distributions to Protect Inference Privacy (2019, neutral): “That’swhyweemployaformula-tionoppositetoL2regularization[38],inordertomakethe magnitudeofnoiseelementsgreater.”