Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift

By Xiang-Yang Li, Shuo Chen, Xiaolin Hu, Jian Yang. 2018

331 citations recorded by the source corpus.

This paper first answers the question ``why do the two most powerful techniques Dropout and Batch Normalization (BN) often lead to a worse performance when they are combined together in many modern neural networks, but cooperate well sometimes as in Wide ResNet (WRN)?'' in both theoretical and empirical aspects. Theoretically, we find that Dropout shifts the variance of a specific neural unit when we transfer the state of that network from training to test. However, BN maintains its statistical variance, which is accumulated from the entire learning procedure, in the test phase. The inconsistency of variances in Dropout and BN (we name this scheme ``variance shift'') causes the unstable numerical behavior in inference that leads to erroneous predictions finally. Meanwhile, the large feature dimension in WRN further reduces the ``variance shift'' to bring benefits to the overall performance. Thorough experiments on representative modern convolutional networks like DenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to the uncovered mechanism, we get better understandings in the combination of these two techniques and summarize guidelines for better practices.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “, 2017) and theoretical (Li et al., 2019) evidence showing that a combination of batch normalization and dropout in Convolutional Neural Networks may have harmful results.”
Meta-Curvature (2019, methodology): “For 5-shot experiments, we also applied dropout to the final linear layer after all convolutional layer and batch normalization layer (Li et al., 2018).”
Dropout vs. batch normalization: an empirical study of their impact to deep learning (2020, methodology): “The low accuracy of dropout combined with batch normalization: contrary to the tests performed here, [17] reported that dropout can be used to improve accuracy.”
Deep learning for cellular image analysis (2019, methodology): “Typically, only one of these methods is used in a model, as performance can suffer if both are used simultaneously 162 .”
SalsaNext: Fast, Uncertainty-Aware Semantic Segmentation of LiDAR Point Clouds (2020, unknown): “ft. Dropout is then placed after the batch normalization. It can, otherwise, result in a shift in the weight distribution which can minimize the batch normalization effect during training as shown in [35]. C. Uncertainty Estimation 1) Heteroscedastic Aleatoric Uncertainty: We can deﬁne aleatoric unc”
Larger Norm More Transferable: An Adaptive Feature Norm Approach for Unsupervised Domain Adaptation (2018, methodology): “Dropout is a widely-used regularization technique in deep neural networks [38, 18].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “Ioffe and Szegedy [8] were the ﬁrst to show the advantages of this stochasticity for the generalization of networks, like dropout [207], [208].”
Bond Risk Premia with Machine Learning∗ (2018, unknown): “As far as dropout goes, it is applied after activation as suggested by Li et al. (2018).”
Towards Understanding Regularization in Batch Normalization (2018, methodology): “The improvement by applying dropout after BN has also been observed by a recent work (Li et al., 2018).”
A cloud detection algorithm for satellite imagery based on deep learning (2019, methodology): “Deep learning has been used for various remote sensing tasks, such as road detection (Zhang et al., 2018), sea-land detection (Li et al., 2018a), and land cover mapping (Karakizi et al., 2018).”
Continual Learning, Fast and Slow (2021, background): “However, preliminary results of such strategies are not promising due to the incompatibility between dropout and batch normalization [75], [76].”