Understanding Batch Normalization

By Johan Bjorck, Carla P. Gomes, Bart Selman. 2018

751 citations recorded by the source corpus.

Batch normalization (BN) is a technique to normalize activations in intermediate layers of deep neural networks. Its tendency to improve accuracy and speed up training have established BN as a favorite technique in deep learning. Yet, despite its enormous success, there remains little consensus on the exact reason and mechanism behind these improvements. In this paper we take a step towards a better understanding of BN, following an empirical approach. We conduct several experiments, and show that BN primarily enables training with larger learning rates, which is the cause for faster convergence and better generalization. For networks without BN we demonstrate how large gradient updates can result in diverging loss and activations growing uncontrollably with network depth, which limits possible learning rates. BN avoids this problem by constantly correcting activations to be zero-mean and of unit standard deviation, which enables larger gradient steps, yields faster convergence and may help bypass sharp local minima. We further show various ways in which gradients and activations of deep unnormalized networks are ill-behaved. We contrast our results against recent findings in random matrix theory, shedding new light on classical initialization schemes and their consequences.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Root Mean Square Layer Normalization (2019, background): “Recent studies have proposed alternative explanations for the success of normalization, such as the uncontrollable growth of layer activations in unnormalized deep networks [5].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Batch normalization also smoothens the loss landscape (Santurkar et al., 2018), which enables stable training with larger learning rates and at larger batch sizes (Bjorck et al., 2018; De & Smith, 2020), and it can have a regularizing effect (Hoffer et al., 2017; Luo et al., 2018).”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “BatchNorm enables faster and more stable training because it enables larger learning rates (Bjorck et al., 2018; Santurkar et al., 2018).”
QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “There is now a general consensus in the machine learning community that normalization is a key ingredient to accurate DNNs [5].”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, background): “First, BN makes the optimization landscape smoother (Santurkar et al., 2018), which allows the optimization of deep neural networks to converge faster and better (Bjorck et al., 2018).”
Architecture Matters in Continual Learning (2022, background): “Batch Normalization (BN) (Ioﬀe and Szegedy, 2015) is a normalization scheme that is shown to increase the convergence speed of the network due to its optimization and generalization beneﬁts (Santurkar et al., 2018; Bjorck et al., 2018).”
Understanding and Improving Layer Normalization (2019, neutral): “chNorm [Santurkar et al., 2018]. However, some recent studies point out that the success of BatchNorm relates to the smoother gradients and has little to do with reducing ICS [Santurkar et al., 2018, Bjorck et al., 2018]. Although these studies provide a pioneering perspective to understand BatchNor”
A Mean Field Theory of Batch Normalization (2019, background): “Bjorck et al. (2018) reached the opposite conclusion as our paper for residual networks with batchnorm, that batchnorm works in this setting because it induces beneficial gradient dynamics and thus allows a much bigger learning rate.”
An Exponential Learning Rate Schedule for Deep Learning (2019, background): “(Bjorck et al., 2018) observe that the higher learning rates enabled by BN empirically improves generalization.”
Batch normalization provably avoids ranks collapse for randomly initialised deep networks (2020, background): “The spectral distribution of products of random matrices with i.i.d. standard Gaussian elements has been studied extensively [7, 12, 21].”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “In contrast, using batch normalization layers typically improves both generalization and convergence speed of deep neural networks (Luo et al., 2019; Bjorck et al., 2018).”
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (2020, background): “We therefore conclude that large learning rates are not the primary beneﬁt of batch normalization in residual networks, contradicting the claims of earlier work [16, 17].”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, methodology): “BN has been extensively used in various network architectures [11, 43, 12, 50, 42, 14] for its benefits in improving both the optimization efficiency [19, 9, 21, 5, 37] and generalization capability [19, 3, 5, 48].”
SCSA: Exploring the Synergistic Effects Between Spatial and Channel Attention (2024, neutral): “Conversely, BN’s sensitivity to batch size can introduce statistical noise [2] when processing multi-semantic information.”
On Feature Normalization and Data Augmentation (2020, methodology): “Studies have shown that this smoothes the optimization landscape [46] and enables larger learning rates [2], which leads to faster convergence in practice.”
Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey (2020, background): “jects with ne detail and also reduces training complexity as well as memory consumption for training. To handle the internal covariate shift in the network, the authors have added batch normalization [109] layer on top of convolutional and de-convolutional layers. The architecture of Deconvnet is sh”
DO-Conv: Depthwise Over-Parameterized Convolutional Layer (2020, background): “Normalization layers have been widely used for improving the training of CNNs, while the reason for their success is still being actively studied [31, 4, 21, 20].”
Positional Normalization (2019, methodology): “…normalization methods was initially attributed to “reducing internal covariate shift” in hidden layers [29, 41], an array of recent studies [2, 3, 5, 25, 48, 60, 69, 77] has provided evidence that BN changes the loss surface and prevents divergence even with large step sizes [5], which…”