$L1$ -Norm Batch Normalization for Efficient Training of Deep Neural Networks

By Shuang Wu, Guoqi Li, Lei Deng, Liu Liu, Dong Wu, Yuan Xie, Luping P. Shi. 2018

128 citations recorded by the source corpus.

Batch normalization (BN) has recently become a standard component for accelerating and improving the training of deep neural networks (DNNs). However, BN brings in additional calculations, consumes more memory, and significantly slows down the training iteration. Furthermore, the nonlinear square and sqrt operations in the normalization process impede low bit-width quantization techniques, which draw much attention to the deep learning hardware community. In this paper, we propose an <inline-formula> <tex-math notation="LaTeX">$L1$ </tex-math></inline-formula>-norm BN (L1BN) with only linear operations in both forward and backward propagations during training. L1BN is approximately equivalent to the conventional <inline-formula> <tex-math notation="LaTeX">$L2$ </tex-math></inline-formula>-norm BN (L2BN) by multiplying a scaling factor that equals <inline-formula> <tex-math notation="LaTeX">$({\pi }/{2})^{1/2}$ </tex-math></inline-formula>. Experiments on various convolutional neural networks and generative adversarial networks reveal that L1BN can maintain the same performance and convergence rate as L2BN but with higher computational efficiency. In real application-specified integrated circuit synthesis with reduced resources, L1BN achieves 25% speedup and 37% energy saving compared to the original L2BN. Our hardware-friendly normalization method not only surpasses L2BN in speed but also simplifies the design of deep learning accelerators. Last but not least, L1BN promises a fully quantized training of DNNs, which empowers future artificial intelligence applications on mobile devices with transfer and continual learning capability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Soham De
2. Samuel L. Smith
3. Quoc V. Le
4. Daniel Soudry
5. Elad Hoffer
6. Itay Hubara
7. Ling Shao
8. Song Han

In Computer Vision and Pattern Recognition

1. Ling Shao
2. Luc Van Gool
3. Dacheng Tao
4. Ming-Hsuan Yang
5. Soham De
6. Quoc V. Le
7. Alan L. Yuille
8. Jitendra Malik

In Electrical and Electronic Engineering

1. Guoqi Li
2. Tailin Liang
3. Yuan Xie
4. Song Han
5. Ling Shao
6. C. John Glossner
7. By Lei Deng
8. Daniel Soudry

In Efficiency & Compression

1. Elad Hoffer
2. Guoqi Li
3. Luping P. Shi
4. Daniel Soudry
5. Itay Hubara
6. Ron Banner
7. Yuan Xie
8. By Lei Deng

In Optimization & Training

1. Soham De
2. Samuel L. Smith
3. Daniel Soudry
4. Elad Hoffer
5. Itay Golan
6. Itay Hubara
7. Guoqi Li
8. Luping P. Shi

Citations in the corpus

• Root Mean Square Layer Normalization (2019, background): “Hoffer et al. [10] and Wu et al. [33] observe that l 1 -norm can act as an alternative of variance in BatchNorm with the benefit of fewer nonlinear operations and higher computational efficiency.”
• Batch Normalization Biases Deep Residual Networks Towards Shallow Paths (2020, background): “The normalization operation itself is expensive, and can constitute a significant fraction of the total cost of computing a parameter update (Wu et al., 2018).”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Therefore, reducing BN’s nonlinearity and quantizing BN operations also matter [180], [194], [195].”
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “To solve this, l1-norm BN [236] has only linear operations in both forward and backward training.”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “Adaptive IN [69] ΠIN (X) ∈ Rmd×hw Standardizing Generated γ, β ∈ R ECCV, 2018 L(1)-Norm BN [91], [92] ΠBN (X) ∈ Rd×mhw Standardizing with L(1)-Norm divided Learnable γ, β ∈ R NeurIPS, 2018”
• Scalable Methods for 8-bit Training of Neural Networks (2018, background): “Recently L1 batch norm with only linear operations in both forward and backward propagation was suggested by [27, 11] with improved numerical stability.”
• Norm matters: efficient and accurate normalization schemes in deep networks (2018, background): “[39], appearing in parallel to us (within a week).”
• Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, background): “Multiple standardization options have been discovered for normalizing mini-batch data, including the L2 standardization [19], the L1-standardization [47, 13] and the L∞-standardization [13].”