Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

By Devansh Arpit, Yingbo Zhou, Bhargava Urala Kota, Venugopal Govindaraju. 2016

131 citations recorded by the source corpus.

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of mean and standard deviation of input (distribution) to hidden layers inaccurate for validation due to shifting parameter values (especially during initial training epochs). Also, BN cannot be used with batch-size 1 during training. We address these drawbacks by proposing a non-adaptive normalization technique for removing internal covariate shift, that we call Normalization Propagation. Our approach does not depend on batch statistics, but rather uses a data-independent parametric estimate of mean and standard-deviation in every layer thus being computationally faster compared with BN. We exploit the observation that the pre-activation before Rectified Linear Units follow Gaussian distribution in deep networks, and that once the first and second order statistics of any given dataset are normalized, we can forward propagate this normalization without the need for recalculating the approximate statistics for hidden layers.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Weight Standardization (2019, methodology): “To explore other dimensions [2, 33], Layer Normalization [3] normalizes data on the channel dimension, Instance Normalization [40] performs BN for each sample individually, and Group Nor-malization [42] ﬁnds a better middle point between Layer Normalization and Instance Normalization.”
Group Normalization (2018, methodology): “moments(x, [2, 3, 4], keep dims=True) x = (x − mean) / tf.”
Spectral Normalization for Generative Adversarial Networks (2018, methodology): “Still another technique to regularize the weight matrix is to use the Frobenius norm: where Originally, these regularization techniques were invented with the goal of improving the generalization performance of supervised training (Salimans & Kingma, 2016; Arpit et al., 2016).”
Thinking Fast and Slow with Deep Learning and Tree Search (2017, methodology): “Different biases are used in each position for all convolution layers and normalisation propagation [18] is applied in layers 1-12.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, neutral): “For ReLUs, γ = √ 2/(1− (1/π)) (Arpit et al., 2016).”
Sharp Minima Can Generalize For Deep Nets (2017, methodology): “Instances of commonly used reparametrization are batch normalization (Ioffe & Szegedy, 2015), or the virtual batch normalization variant (Salimans et al., 2016), and weight normalization (Badrinarayanan et al., 2015; Salimans & Kingma, 2016; Arpit et al., 2016).”
Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models (2017, methodology): “However, unlike batchnorm which can be easily applied to an existing model, these methods may require careful analysis of nonlinearities [1] and may change the class of functions representable by the model [2].”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “We experimented with several different normalization schemes including WeightNorm [28], LayerNorm [2] and NormProp [1] but found them to be unstable in training.”
Adding Gradient Noise Improves Learning for Very Deep Networks (2015, methodology): “Similarly, batch normalization and related methods (Ioffe & Szegedy, 2015; Arpit et al., 2016; Salimans & Kingma, 2016), natural gradient descent (Amari, 1998; Desjardins et al.”
Revisiting Small Batch Training for Deep Neural Networks (2018, background): “Different types of normalization have also been proposed (Ba et al., 2016; Salimans & Kingma, 2016; Arpit et al., 2016; Ioffe, 2017; Wu & He, 2018).”
Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks (2017, methodology): “To overcome this problem, normalization propagation (Arpit et al., 2016) uses a data-independent parametric estimate of mean and standard deviation, while layer normalization (Ba et al.”
On the Effects of Batch and Weight Normalization in Generative Adversarial Networks (2017, methodology): “In [2], the authors attempt to solve this problem by enforcing a zero-mean, unit-variance distribution throughout the network.”
On the effect of Batch Normalization and Weight Normalization in Generative Adversarial Networks (2017, methodology): “The detail of our version of Weight Normalization is slightly different to both [12] and [2], which will be explained in section 3.”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “Inspired by the solutions adopted in StyleGAN2 [32] and other works that have sought alternatives to explicit normalization [1, 6, 37], we choose to modify the network so that individual layers and pathways preserve the activation magnitudes on expectation , with the goal of removing or at least…”
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization (2018, background): “…has promoted its key idea that normalizing the inner layers of a neural network stabilizes training which recently led to the development of many such normalization methods such as (Arpit et al., 2016; Klambauer et al., 2017; Salimans and Kingma, 2016) and (Ba et al., 2016) to name just a few.”
Residual Connections Encourage Iterative Inference (2017, methodology): “To address this problem, solutions like better activations (ReLU Nair & Hinton (2010)), weight initialization methods Glorot & Bengio (2010); He et al. (2015) and normalization methods Ioffe & Szegedy (2015); Arpit et al. (2016) have been proposed.”
Unreproducible Research is Reproducible (2019, background): “…of most deep learning paper would not be enough to classify them as empirical research. led to a significant number of works (Salimans & Kingma, 2016; Ba et al., 2016; Cooijmans et al., 2016; Arpit et al., 2016) that focused on normalizations, ahead of a good understanding of why Batch-Norm works.”
Centered Weight Normalization in Accelerating Training of Deep Neural Networks (2017, methodology): “Besides, during test the means and variances are estimated based on the moving averages of mini-batch means and variances [16, 3, 4].”
The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks (2018, unknown): “Debiasing follows the example of Arpit et al. (2016).”