Batch Normalization Biases Deep Residual Networks Towards Shallow Paths
By Soham De, Samuel L. Smith. 2020
20 citations recorded by the source corpus.
Batch normalization has multiple benefits. It improves the conditioning of the loss landscape, and is a surprisingly effective regularizer. However, the most important benefit of batch normalization arises in residual networks, where it dramatically increases the largest trainable depth. We identify the origin of this benefit: At initialization, batch normalization downscales the residual branch relative to the skip connection, by a normalizing factor proportional to the square root of the network depth. This ensures that, early in training, the function computed by deep normalized residual networks is dominated by shallow paths with well-behaved gradients. We use this insight to develop a simple initialization scheme which can train very deep residual networks without normalization. We also clarify that, although batch normalization does enable stable training with larger learning rates, this benefit is only useful when one wishes to parallelize training over large batch sizes. Our results help isolate the distinct benefits of batch normalization in different architectures.
Subfields: Artificial Intelligence, Model Architectures, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Hanxiao Liu
- Karen Simonyan
- Andrew Brock
- Jianfeng Gao
- Mohit Bansal
- Xiangyang Xue
- Ruslan Salakhutdinov
In Model Architectures
- Quoc V. Le
- Hanxiao Liu
- Andrew Brock
- Karen Simonyan
- Frank Hutter
- Changhu Wang
- Alan L. Yuille
- Song Han
In Optimization & Training
- Hanxiao Liu
- Quoc V. Le
- Karen Simonyan
- Andrew Brock
- Xiangyang Xue
- Bhiksha Raj
- Changhu Wang
- Mingxing Tan
Citations in the corpus
- Is normalization indispensable for training deep neural network? (2020, methodology): “Two recent studies, Fixup Init[40] and SkipInit [7] are based on this observation.”
- Evolving Normalization-Activation Layers (2020, background): “Our work is also related to efforts on improving the initialization conditions for deep networks (Zhang et al., 2019; De & Smith, 2020; Bachlechner et al., 2020) in terms of challenging the necessity of traditional normalization layers.”