Norm matters: efficient and accurate normalization schemes in deep networks

By Elad Hoffer, Ron Banner, Itay Golan, Daniel Soudry. 2018

191 citations recorded by the source corpus.

Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present a novel view on the purpose and function of normalization methods and weight-decay, as tools to decouple weights' norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments. We suggest several alternatives to the widely used $L^2$ batch-norm, using normalization in $L^1$ and $L^\infty$ spaces that can substantially improve numerical stability in low-precision implementations as well as provide computational and memory benefits. We demonstrate that such methods enable the first batch-norm alternative to work for half-precision implementations. Finally, we suggest a modification to weight-normalization, which improves its performance on large-scale tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

Root Mean Square Layer Normalization (2019, background): “Hoffer et al. [10] and Wu et al. [33] observe that l 1 -norm can act as an alternative of variance in BatchNorm with the beneﬁt of fewer nonlinear operations and higher computational efﬁciency.”
Initialization and Regularization of Factorized Neural Layers (2021, background): “Recently, Zhang et al. (2019) extended the analysis of Hoffer et al. (2018) to argue that the effective step-size of the weight direction ˆ W is roughly η {} W } 2 F , where η is the SGD step-size.”
Fixup Initialization: Residual Learning Without Normalization (2019, methodology): “…surface may be alleviated for residual networks by using smaller initialization than the standard ones such as Xavier or He’s initialization in residual branches. van Laarhoven (2017); Hoffer et al. (2018) studied the effect of (batch) normalization and weight decay on the effective learning rate.”
Three Mechanisms of Weight Decay Regularization (2018, background): “As shown by Hoffer et al. (2018), the weight direction ˆ θ l = θ l / (cid:107) θ l (cid:107) 2 is updated according to ) .”
An Exponential Learning Rate Schedule for Deep Learning (2019, background): “However, Hoffer et al. (2018b) shows that ﬁxing the output layer randomly doesn’t harm the performance of the network.”
Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory (2019, background): “Hoffer et al. (2018) further study the relationship between weight decay and batch normalization, and they develop normalization with respect to other norms.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Therefore, reducing BN’s nonlinearity and quantizing BN operations also matter [180], [194], [195].”
Positional Normalization (2019, methodology): “…methods was initially attributed to “reducing internal covariate shift” in hidden layers [29, 41], an array of recent studies [2, 3, 5, 25, 48, 60, 69, 77] has provided evidence that BN changes the loss surface and prevents divergence even with large step sizes [5], which accelerates…”
Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks (2019, background): “Salimans & Kingma (2016); Hoffer et al. (2018) direction from magnitude to accelerate the training procedure.”
Evolving Normalization-Activation Layers (2020, methodology): “This observation is aligned with some previous ﬁndings that scale invariance might play a useful role from the optimization point of view (Hoffer et al., 2018; Li & Arora, 2020). γ + β The numerator of EvoNorm-S0 resembles the Swish activation function (Ramachandran et al., 2018) and its…”
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence (2019, background): “Several authors [38, 15, 35] have observed that WD increases the effective learning rate ⌘eff,t = ⌘t/kwtk(2)2, where ⌘t is the learning rate at epoch t and kwtk(2)2 is the squared-norm of weights at epoch t, by decreasing the weight norm, which increases the effective gradient noise, which promotes ”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, background): “Multiple standardization options have been discovered for normalizing mini-batch data, including the L2 standardization [19], the L1-standardization [47, 13] and the L∞-standardization [13].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “Note that L normalization is made equivalent to L normalization (under mild assumptions) by multiplying it by a scaling factor √ π 2 [91], [92].”
Large-Scale Differentially Private BERT (2021, background): “This has been noted in the literature (Hoffer et al., 2019; Cho and Lee, 2017; Davody et al., 2020; Heo et al., 2021), and Heo et al. (2021) propose a modiﬁcation to Adam to handle these layers for non-private training, and Davody et al. (2020) introduce batch normalization layers to make networks…”
Four Things Everyone Should Know to Improve Batch Normalization (2019, background): “However, related work [8, 32] has shown that weight decay actually has a subtle effect on the effective learning rate of networks trained with Batch Normalization — without weight decay, the weights in a batch-normalized network grow to have large magnitudes, which has an inverse effect on the effec”
FixNorm: Dissecting Weight Decay for Training Deep Neural Networks (2021, methodology): “Hoffer et al. (2018) shows that the performance can be recovered when training without weight decay by applying an LR correction technique.”
Scalable Methods for 8-bit Training of Neural Networks (2018, background): “Recently L1 batch norm with only linear operations in both forward and backward propagation was suggested by [27, 11] with improved numerical stability.”