Towards Understanding Regularization in Batch Normalization

By Ping Luo, Xinjiang Wang, Wenqi Shao, Zhanglin Peng. 2018

195 citations recorded by the source corpus.

Batch Normalization (BN) improves both convergence and generalization in training neural networks. This work understands these phenomena theoretically. We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function. This basic network helps us understand the impacts of BN in three aspects. First, by viewing BN as an implicit regularizer, BN can be decomposed into population normalization (PN) and gamma decay as an explicit regularization. Second, learning dynamics of BN and the regularization show that training converged with large maximum and effective learning rate. Third, generalization of BN is explored by using statistical mechanics. Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

Weight Standardization (2019, background): “[27] focuses on the generalization properties of BN by studying its regularization effects.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Batch normalization also smoothens the loss landscape (Santurkar et al., 2018), which enables stable training with larger learning rates and at larger batch sizes (Bjorck et al., 2018; De & Smith, 2020), and it can have a regularizing effect (Hoffer et al., 2017; Luo et al., 2018).”
Quantifying Generalization in Reinforcement Learning (2018, background): “Batch normalization (Ioﬀe and Szegedy, 2015) is known to have a substantial regularizing eﬀect in supervised learning (Luo et al., 2018).”
Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise (2019, background): “BN with the estimation noise can work as an adaptive regularizer of parameters (Luo et al. 2019; Shekhovtsov and Flach 2019) and the moderate noise can help escape bad local minima and saddle points (Jin et al.”
GemNet: Universal Directional Graph Neural Networks for Molecules (2021, methodology): “The variance of activations in a model is usually stabilized using normalization methods, which has various positive effects on training [16, 41, 52].”
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization (2021, background): “Luo et al. (2018) investigated an explicit regularization form of BN such that improving the robustness of optimization.”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “In contrast, using batch normalization layers typically improves both generalization and convergence speed of deep neural networks (Luo et al., 2019; Bjorck et al., 2018).”
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs (2020, methodology): “…the role and expressive power of the afﬁne parameters in BatchNorm, which typically receive less research attention than the normalization aspect (Santurkar et al., 2018; Bjorck et al., 2018; Yang et al., 2019; Luo et al., 2019 these parameters have relatively little effect on accuracy (Figure 1).”
Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, background): “This is likely caused by the known regularization effect of BatchNorm (Hoffer et al., 2017; Luo et al., 2019; De & Smith, 2020).”
A systematic review on overfitting control in shallow and deep neural networks (2021, background): “In Luo et al. (2019), it is shown that this scheme can control overfitting problems.”
Dive into Deep Learning (2020, background): “, 2018) and (Luo et al., 2018) relate the properties of batch normalization to Bayesian priors and penalties respectively.”
Positional Normalization (2019, methodology): “…methods was initially attributed to “reducing internal covariate shift” in hidden layers [29, 41], an array of recent studies [2, 3, 5, 25, 48, 60, 69, 77] has provided evidence that BN changes the loss surface and prevents divergence even with large step sizes [5], which accelerates…”
Deep Learning for Generic Object Detection: A Survey (2018, background): “ing the design of speciﬁc deep model suitable for object detection. In the research community, training from scratch becomes possible, as more training data and better training strategy are available [278,179,178]. More backbone deep models will be speciﬁcally designed for object detection. (3) Impr”
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks (2020, background): “[33] analyzed the regularization benefits of batch normalization.”
Batch Normalization Biases Deep Residual Networks Towards Shallow Paths (2020, methodology): “Some authors have observed that batch normalization has a regularizing effect (Hoffer et al., 2017; Luo et al., 2019).”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, background): “This breaks the regularization effect of BN (Luo et al., 2018).”
Is normalization indispensable for training deep neural network? (2020, background): “Other studies [29, 22, 4, 38] also show several other advantages provided by normalization in terms of loss landscape, regularization, etc.”
Dropout vs. batch normalization: an empirical study of their impact to deep learning (2020, background): “Two places to start are increasing the batch size, as recommended in [20], and reducing the dropout rate, as recommended in [17].”
Differentiable Learning-to-Normalize via Switchable Normalization (2018, methodology): “However, for small minibatch, BKN reported 76.1% that was evaluated in a micro-batch setting where 256 samples are used to compute gradients and 4 samples to estimate the statistics.”