Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

By Soham De, Samuel L. Smith. 2020

166 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “Batch normalization also smoothens the loss landscape (Santurkar et al., 2018), which enables stable training with larger learning rates and at larger batch sizes (Bjorck et al., 2018; De & Smith, 2020), and it can have a regularizing effect (Hoffer et al., 2017; Luo et al., 2018).”
Generating Images with Sparse Representations (2021, methodology): “…the original architecture that we found to improve stability, training speed, and memory consumption: ReZero We use ReZero (Bachlechner et al., 2020; De & Smith, 2020), multiplying each residual connection with a zero-initialized scalar value, which is optimized jointly with the model parameters.”
Going deeper with Image Transformers (2021, methodology): “As pointed in our paper, in that respect our LayerScale approach is more related to Rezero [2] and Skipinit [16], Fixup [76], and T-Fixup [34].”
Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, methodology): “…growth arises because, at initialization, the variance of the activations satisfy Var( x (cid:96) +1 ) = Var( x (cid:96) ) + Var( f (cid:96) ( x (cid:96) )) , while BatchNorm ensures that the variance of the activations at the end of each residual branch is independent of depth (De & Smith, 2020).”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, methodology): “While using the “post-norm” option of adding the normalization layer after the skip and residual branches typically improves performance, we stick to this design due to this architecture being more scalable in general (De and Smith, 2020).”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, methodology): “A range of other alternatives to batch normalization have recently been identiﬁed in non-private deep learning (Brock et al., 2021a; De and Smith, 2020; Kolesnikov et al., 2020; Zhang et al., 2018b).”
On the Generalization Benefit of Noise in Stochastic Gradient Descent (2020, unknown): “We train both with batch normalization and without batch normalization (using Regularized SkipInit (De & Smith, 2020)), and we provide the optimal learning rates with batch normalization in figure 2(c), and the optimal learning rates without batch normalization in figure 2(d).”
Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, methodology): “Note, by contrast, performance degrades for very large batch sizes in standard pipelines [Smith et al., 2020].”
TransNeXt: Robust Foveal Visual Perception for Vision Transformers (2023, unknown): “From the perspective of biological vision, the current design of pixel-focused attention can effectively simulate the highly complex transsaccadic perception in human visual perception.”
GemNet: Universal Directional Graph Neural Networks for Molecules (2021, methodology): “We found that other measures such as adaptive gradient clipping [7], scaled weight standardization [6], or weighting the residual block with zero at initialization [16] are not beneficial for model accuracy.”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, methodology): “Although ResNets seem to overcome the dilemma, managing to train a network with hundreds of layers, some works [25, 91] indicate ResNets might not be as deep as they appear to be.”
Simple Baselines for Image Restoration (2022, methodology): “In addition, we apply skip-init[10] to stabilize training following [22].”