Group Normalization

By Yuxin Wu, Kaiming He. 2018

4,460 citations recorded by the source corpus.

Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems—BN’s error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN’s usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN’s computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform its BN-based counterparts for object detection and segmentation in COCO (https://github.com/facebookresearch/Detectron/blob/master/projects/GN), and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, positive): “For the baseline CNNs, we use ResNet (He et al., 2016), but replace the Batch Normalization layers (Ioffe & Szegedy, 2015) with Group Normalization (Wu & He, 2018), and used standardized convolutions (Qiao et al., 2019).”
Denoising Diffusion Probabilistic Models (2020, methodology): “To represent the reverse process, we use a U-Net backbone similar to an unmasked PixelCNN++ [49, 45] with group normalization throughout [62].”
Rethinking ImageNet Pre-Training (2018, methodology): “Successful forms of normalization include normalized parameter initialization [11, 16] and activation normalization layers [20, 2, 46, 48].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “To avoid mismatch in batch statistics between the two data sources, in the co-training experiments we replace Batch-Norm with GroupNorm [62] and also standardize [45] the weights of the convolutions.”
Weight Standardization (2019, methodology): “Although some normalization methods are specifically designed for micro-batch training such as Group Nor-malization (GN) [42], they still have difﬁculty matching the performances of BN in large-batch training (Fig.”
Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “These upgrades, developed primarily in [39, 18, 13], allow the model to be trained effectively from random initialization, thus enabling a meaningful from-scratch baseline.”
Cascade R-CNN: High Quality Object Detection and Instance Segmentation (2019, methodology): “Some more recent works have focused on normalizing feature statistics [44], [55], modeling relations between instances [27], non maximum suppression (NMS) [1], and other aspects [38], [49].”
Panoptic Feature Pyramid Networks (2019, methodology): “gure3. Starting from the deepest FPN level (at 1/32 scale), we perform three upsampling stages to yield a feature map at 1/4 scale, where each upsampling stage consists of 3 3 convolution, group norm [54], ReLU, and 2 bilinear upsampling. This strategy is repeated for FPN scales 1/16, 1/8, and 1/4 (”
VQ-DRAW: A Sequential Discrete VAE (2020, methodology): “In most of my CNN models, I use group normalization [10] after every ReLU non-linearity.”
Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “We also experiment with a layer [43] that we refer to as adaptive group normalization (AdaGN), which incorporates the timestep and class embedding into each residual block after a group normalization operation [69], similar to adaptive instance norm [27] and FiLM [48].”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, neutral): “Other popular normalization layers include the Instance Normalization (IN) [43], Fayer Normalization (FN) [2], Group Normalization (GN) [47], and Weight Normalization (WN) [42].”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, background): “Each layer contains a convolution, followed by dropout, group normalization with a single group (Wu & He, 2018) and a ReLU non-linearity.”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “We normalize both across the feature and temporal dimension for each sample which is equivalent to group normalization with a single normalization group (Wu & He, 2018).”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Although a number of alternative normalizers have been proposed (Ba et al., 2016; Wu & He, 2018; Huang et al., 2020), these alternatives often achieve inferior test accuracies and introduce their own disadvantages, such as additional compute costs at inference.”