Normalization Techniques in Training DNNs: Methodology, Analysis and Application
By Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, Ling Shao. 2020
457 citations recorded by the source corpus.
Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Soham De
- Steven C. H. Hoi
- Samuel L. Smith
- Andrew Brock
- Andrew Zisserman
- Karen Simonyan
- Mohit Bansal
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Steven C. H. Hoi
- Andrew Zisserman
- Andrea Vedaldi
In Optimization & Training
- Soham De
- Andrew Brock
- Samuel L. Smith
- Karen Simonyan
- Steven C. H. Hoi
- Timo Aila
- Miika Aittala
- Chenghao Liu
Citations in the corpus
- High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Although a number of alternative normalizers have been proposed (Ba et al., 2016; Wu & He, 2018; Huang et al., 2020), these alternatives often achieve inferior test accuracies and introduce their own disadvantages, such as additional compute costs at inference.”
- Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “For example, it is advisable to keep input and output signal magnitudes fixed to, e.g., unit variance, and to avoid large variation in gradient magnitudes on a per-sample basis [5, 21].”
- Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, methodology): “In standard backbone networks such as ResNets (He et al., 2016), it is natural to use Batch Normalization (BN) (Ioffe & Szegedy, 2015), which has enabled the deep learning community to make substantial progress in many applications (Huang et al., 2020).”
- Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, background): “These include LayerNorm (Ba et al., 2016), InstanceNorm (Ulyanov et al., 2016), GroupNorm (Wu & He, 2018), and many more (Huang et al., 2020).”