Decorrelated Batch Normalization
By Lei Huang, Dawei Yang, Bo Lang, Jia Deng. 2018
217 citations recorded by the source corpus.
Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find that PCA whitening causes a problem we call stochastic axis swapping, which is detrimental to learning. We show that ZCA whitening does not suffer from this problem, permitting successful learning. DBN retains the desirable qualities of BN and further improves BN's optimization efficiency and generalization ability. We design comprehensive experiments to show that DBN can improve the performance of BN on multilayer perceptrons and convolutional neural networks. Furthermore, we consistently improve the accuracy of residual networks on CIFAR-10, CIFAR-100, and ImageNet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “For example, [3, 14, 35] rely on feature decorrelation to solve the mode collapse problem in self-supervised learning when negative pairs are missing, and [5,15,34] use decorrelation to improve neural networks’ features and hence boost generalization.”
- RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, methodology): “To alleviate these problems, previous studies have shown that the goal of WT can be achieved without the eigen-decomposition through the whitening loss [7] or approximating the whitening transformation matrix using Newton’s iteration [21, 20, 43].”
- Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, methodology): “Additionally, we also use the extra learnable parameters γ and β, as in previous normalization methods [19, 3, 17, 48], since normalizing the activations constrains the model’s capacity for representation.”
- Whitening for Self-Supervised Representation Learning (2020, unknown): “Note that Huang et al. (2018); Siarohin et al.”
- Four Things Everyone Should Know to Improve Batch Normalization (2019, methodology): “ence paper at ICLR 2020 parameterizing them in terms of a vector giving the direction of the weights and an explicit scale, which must be initialized very carefully. Decorrelated Batch Normalization (Huang et al., 2018) performs ZCA whitening in its normalization layer, and Iterative Normalization (”
- Differentiable Learning-to-Normalize via Switchable Normalization (2018, background): “between standardization and whitening to improve a wide range of whitened neural networks [3, 21, 20, 9], which decorrelate the hidden representation [9, 21] or the backpropagated errors [20], enabling SGD to mimic natural gradient descent.”
- Whitening and Coloring Batch Transform for GANs (2018, methodology): “(2018)), making the gradient-descent weight updates closer to Newton updates (Huang et al. (2018); LeCun et al.”
- On Feature Decorrelation in Self-Supervised Learning (2021, methodology): “As a sanity check, we adapt DBN [25] to standardize the covariance matrix for mitigation of this issue.”
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “Decorrelated BN [34] ΠBN (X) ∈ Rd×mhw ZCA whitening Learnable γ, β ∈ R CVPR, 2018 Group normalization (GN) [22] ΠGN (X) ∈ Rmg×shw Standardizing Learnable γ, β ∈ R ECCV, 2018”