Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
By Sergey Ioffe, Christian Szegedy. International Conference on Machine Learning, 2015
47,277 citations recorded by the source corpus.
Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.
Subfields: Artificial Intelligence, Model Architectures, Optimization & Training
View Paper: https://arxiv.org/abs/1502.03167
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
- Dacheng Tao
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Alan L. Yuille
- Ruslan Salakhutdinov
- Kaiming He
- Dacheng Tao
- Jianfeng Gao
- Kyunghyun Cho
In Optimization & Training
- Yoshua Bengio
- Quoc V. Le
- Peter Richtárik
- Frank Hutter
- Martin Jaggi
- Simon Lacoste-Julien
- Sebastian U. Stich
- Sergey Levine
Citations in the corpus
- Deep Residual Learning for Image Recognition (2015, methodology): “We do not use dropout [14], following the practice in [16].”
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “…example: M batchwise ( X, m ) j,i = 1 if X j,i is in the top m values for to expert i 0 otherwise (18) As our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as M batchwise ) requires modifications to the inference when we may not…”
- Neural Machine Translation in Linear Time (2016, methodology): “We introduce a modification to Batch Normalization (BN) (Ioffe and Szegedy, 2015) in order to make it applicable to target networks and decoders.”
- Layer Normalization (2016, background): “Recently, batch normalization [Ioffe and Szegedy, 2015] has been proposed to reduce training time by including additional normalization stages in deep neural networks.”
- Rethinking the Inception Architecture for Computer Vision (2015, methodology): “This is supported by the fact that the main classifier of the network performs better if the side branch is batch-normalized [7] or has a dropout layer.”
- Universal Language Model Fine-tuning for Text Classification (2018, methodology): “Following standard practice for CV classifiers, each block uses batch normalization (Ioffe and Szegedy, 2015) and dropout, with ReLU activations for the intermediate layer and a softmax activation that outputs a probability distribution over target classes at the last layer.”
- Learned in Translation: Contextualized Word Vectors (2017, methodology): “…representation for all inputs. x pool = max( X | y ); mean( X | y ); min( X | y ); x self (15) We feed this joined representation through a three-layer, batch-normalized [Ioffe and Szegedy, 2015] maxout network [Goodfellow et al., 2013] to produce a probability distribution over possible classes.”
- Object Detection Networks on Convolutional Feature Maps (2015, methodology): “Regarding GoogleNet, we train the 426 BN-Inception model [32] on ImageNet classification.”
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, methodology): “4Concurrently, a Batch Normalization method [15] achieves results on par with ours: 5.”
- InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “[4] introduced the Generative Adversarial Networks (GAN), a framework for training deep generative models using a minimax game.”
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “For the baseline CNNs, we use ResNet (He et al., 2016), but replace the Batch Normalization layers (Ioffe & Szegedy, 2015) with Group Normalization (Wu & He, 2018), and used standardized convolutions (Qiao et al., 2019).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “We modified the base ResNet-50 with the anti-alias improvements from (Zhang, 2019) and used weight norm (Salimans & Kingma, 2016) instead of batch norm (Ioffe & Szegedy, 2015) to avoid leaking information about duplicates via batch statistics - a problem previously noted in (Henaff, 2020).”
- Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “Following [15], we adopt an extra BatchNorm layer [31] without affine transformation ( affine=False ).”
- Continuous control with deep reinforcement learning (2015, methodology): “We address this issue by adapting a recent technique from deep learning called batch normalization [7].”
- A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “The other type of augmentation involves appearance transformation, such as color distortion (including color dropping, brightness, contrast, saturation, hue) (Howard, 2013; Szegedy et al., 2015), Gaussian blur, and Sobel filtering.”
- Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “Our encoders f q and f k both have Batch Normalization (BN) [37] as in the standard ResNet [33].”