How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift)

By Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, Aleksander Madry. 2018

1,738 citations recorded by the source corpus.

Batch Normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks (DNNs). Despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood. The popular belief is that this effectiveness stems from controlling the change of the layers' input distributions during training to reduce the so-called "internal covariate shift". In this work, we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother. This smoothness induces a more predictive and stable behavior of the gradients, allowing for faster training. These findings bring us closer to a true understanding of our DNN training toolkit.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “…include the debate on ﬂat vs sharp minima [16, 5, 29, 15], explanations of the efﬁcacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6], qualitative studies and visualizations of the loss surface [11], and…”
Root Mean Square Layer Normalization (2019, background): “[23] who argue that the success of normalization methods does not come from the added stability to layer inputs, but due to increased smoothness of the optimization landscape).”
Weight Standardization (2019, methodology): “It has been demonstrated that BN inﬂuences network training in a fundamental way: it makes the landscape of the optimization problem signiﬁcantly smoother [37].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Batch normalization also smoothens the loss landscape (Santurkar et al., 2018), which enables stable training with larger learning rates and at larger batch sizes (Bjorck et al., 2018; De & Smith, 2020), and it can have a regularizing effect (Hoffer et al., 2017; Luo et al., 2018).”
Squeeze-and-Excitation Networks (2017, methodology): “By regulating the distribution of the inputs to each layer, Batch Normalization (BN) [6] added stability to the learning process in deep networks and produced smoother optimisation surfaces [12].”
Fixup Initialization: Residual Learning Without Normalization (2019, background): “Our analysis in Section 2 corroborates the latter idea of Santurkar et al. (2018) by showing that standard initialization leads to very steep loss surface at initialization.”
How to train your MAML (2018, background): “Using running statistics instead of batch statistics, can greatly increase convergence speed, stability and generalization performance as the normalized features will result in smoother optimization landscape Santurkar et al. (2018).”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “BatchNorm enables faster and more stable training because it enables larger learning rates (Bjorck et al., 2018; Santurkar et al., 2018).”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “First, BN makes the optimization landscape smoother (Santurkar et al., 2018), which allows the optimization of deep neural networks to converge faster and better (Bjorck et al., 2018).”
Architecture Matters in Continual Learning (2022, background): “Batch Normalization (BN) (Ioﬀe and Szegedy, 2015) is a normalization scheme that is shown to increase the convergence speed of the network due to its optimization and generalization beneﬁts (Santurkar et al., 2018; Bjorck et al., 2018).”
Understanding and Improving Layer Normalization (2019, background): “However, some recent studies point out that the success of BatchNorm relates to the smoother gradients and has little to do with reducing ICS [Santurkar et al., 2018, Bjorck et al., 2018].”
Troubling Trends in Machine Learning Scholarship (2018, background): “normalization may be off the mark [65].”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, background): “However, [8] shows that batch normalization’s success has little to do with covariate shift, but comes instead from smoothing the loss landscape.”
Defining Benchmarks for Continual Few-Shot Learning (2020, background): “Another example are Relational Networks (Santoro et al., 2017), originally created to deal with relational reasoning; they have been adapted to the few-shot learning setting with good performance (Santurkar et al., 2018).”
An Exponential Learning Rate Schedule for Deep Learning (2019, methodology): “Santurkar et al. (2018) observed similar phenomenon in experiments, which they described as a smoothening of the objective due to BN. 2.”
Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification (2020, background): “This suggests that the BN operation enhances the predictive and stable behavior of the training process [33], which is more suitable for VI-ReID with abundant noise.”
SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection (2019, background): “BN has been proved to be an effective solution of the problem through significantly smoothing the optimization landscape [49, 63].”
Ultra-Low Precision 4-bit Training of Deep Neural Networks (2020, background): “[31] demonstrated that instead of suppressing ICS, batchnorm layers adjust mean and variance to compensate its effects.”