Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

By Sergey Ioffe. 2017

595 citations recorded by the source corpus.

Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Rethinking ImageNet Pre-Training (2018, background): “This reduces batch sizes as constrained by memory, and small batch sizes severely degrade the accuracy of BN [19, 34, 48].”
Big Transfer (BiT): General Visual Representation Learning (2019, background): “However, BN performs worse when the number of images on each accelerator is too low [15].”
Weight Standardization (2019, background): “To alleviates BN’s issue when the batches become small, BR [16] constrains the statistics of BN within a certain range to reduce their drift when the batch size is not suf-ﬁciently large, and [30] proposes synchronized BN which synchronizes statistics across multiple GPUs through engi-2 neering.”
Group Normalization (2018, background): “Comparison with Batch Renorm (BR) BR (Ioffe 2017) introduces two extra parameters [r and d in Ioffe (2017)] that constrain the estimated mean and variance of BN.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “In MoCo v3, all layers in both MLPs have BN [22], following SimCLR [9].”
Unified Perceptual Parsing for Scene Understanding (2018, neutral): “It is worth noting that batch size has proven to be important to generate accurate statistics for tasks like classification [38], semantic segmentation [16] and object detection [39].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Finally, the performance of batch normalization is sensitive to the batch size, and batch normalized networks perform poorly when the batch size is too small (Hoffer et al., 2017; Ioffe, 2017; Wu & He, 2018), which limits the maximum model size we can train on finite hardware.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “Batch renormalization [21] was recently proposed to address this issue by gradually using popular statistics during training.”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “Batch Renormalization (Ioffe, 2017).”
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, background): “data [35], and we do not want this effect to obscure the investigated behavior.”
Real-Time Federated Evolutionary Neural Architecture Search (2020, background): “Apart from this, the trainable parameters in the batch normalization layer may slow down the convergence speed in federated learning, since they perform poorly for learning with small batch sizes [53] and weight sharing training paradigms [39], [54], [55], especially for non-IID scenarios.”
Data Augmentation Generative Adversarial Networks (2017, methodology): “The UResNet generator has a total of 8 blocks, each block having 4 convolutional layers (with leaky rectiﬁed linear (relu) activations and batch renormalisation (batchrenorm) (Ioffe, 2017)) followed by one downscaling or upscaling layer.”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, methodology): “Batch Renormalization Ioffe (2017) was proposed to alleviate the non-i.i.d or small batches in the training data.”
Latent Replay for Real-Time Continual Learning (2019, methodology): “In the context of latent replay, if we leave the BRN moments free to adapt, the activations stored in the external memory suffer the aging effect described in Section IV.”
A ConvNet for the 2020s (2022, background): “Substituting BN with LN. BatchNorm [38] is an essential component in ConvNets as it improves the convergence and reduces overﬁtting.”