Iterative Normalization: Beyond Standardization Towards Efficient Whitening
By Lei Huang, Yi Zhou, Fan Zhu, Li Liu, Ling Shao. 2019
188 citations recorded by the source corpus.
Batch Normalization (BN) is ubiquitously employed for accelerating neural network training and improving the generalization capability by performing standardization within mini-batches. Decorrelated Batch Normalization (DBN) further boosts the above effectiveness by whitening. However, DBN relies heavily on either a large batch size, or eigen-decomposition that suffers from poor efficiency on GPUs. We propose Iterative Normalization (IterNorm), which employs Newton’s iterations for much more efficient whitening, while simultaneously avoiding the eigen-decomposition. Furthermore, we develop a comprehensive study to show IterNorm has better trade-off between optimization and generalization, with theoretical and experimental support. To this end, we exclusively introduce Stochastic Normalization Disturbance (SND), which measures the inherent stochastic uncertainty of samples when applied to normalization operations. With the support of SND, we provide natural explanations to several phenomena from the perspective of optimization, e.g., why group-wise whitening of DBN generally outperforms full-whitening and why the accuracy of BN degenerates with reduced batch sizes. We demonstrate the consistently improved performance of IterNorm with extensive experiments on CIFAR-10 and ImageNet over BN and DBN.
Subfields: Artificial Intelligence, Atomic and Molecular Physics, and Optics, Industrial and Manufacturing Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Atomic and Molecular Physics, and Optics
In Industrial and Manufacturing Engineering
Citations in the corpus
- Concept whitening for interpretable image recognition (2020, methodology): “…up back-propagation is different from that for concept alignment–the rotation matrix is not present in other work on whitening, nor is the notion of a concept–however, we can leverage some of the optimization tools used in that work on whitening (Huang et al., 2019; 2018a; Siarohin et al., 2018).”
- RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, methodology): “For fair comparisons with other normalization techniques, we re-implement IBN-Net [44] and IterNorm [22] on our baseline models and compare them with our methods.”
- Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise (2019, methodology): “First, the added-in module extracts internal information of a networks which can be squeezed channel-wise information (Hu, Shen, and Sun 2018; Li et al. 2019; Huang et al. 2019) or spatial information (Wang et al. 2018; Li, Hu, and Yang 2019).”
- Four Things Everyone Should Know to Improve Batch Normalization (2019, methodology): “, 2018) performs ZCA whitening in its normalization layer, and Iterative Normalization (Huang et al., 2019) makes it more efficient via a Newton iteration approach.”
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “proposed iterative normalization (IterNorm) [81] to improve the computational efficiency and numerical stability of ZCA whitening, since it can avoid eigen-decomposition or SVD by employing Newton’s iteration for approximating the whitening matrix Σ− 1 2 .”
- On Feature Decorrelation in Self-Supervised Learning (2021, background): “Meanwhile, feature decorrelation appears to be a valuable idea in the field of machine learning: In discriminative tasks, [12, 48] introduce in objective functions additional terms regularizing correlation matrices and [25, 26] develop normalization layers standardizing covariance matrices to obtain”