Weight Standardization

By Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan L. Yuille. 2019

148 citations recorded by the source corpus.

Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, i.e., each GPU typically has only 1-2 images for training, which is inevitable for many computer vision tasks, e.g., object detection and semantic segmentation, constrained by memory consumption. To address this issue, we propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to bring two success factors of BN into micro-batch training: 1) the smoothing effects on the loss landscape and 2) the ability to avoid harmful elimination singularities along the training trajectory. WS standardizes the weights in convolutional layers to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients; BCN combines batch and channel normalizations and leverages estimated statistics of the activations in convolutional layers to keep networks away from elimination singularities. We validate WS and BCN on comprehensive computer vision tasks, including image classification, object detection, instance segmentation, video recognition and semantic segmentation. All experimental results consistently show that WS and BCN improve micro-batch training significantly. Moreover, using WS and BCN with micro-batch training is even able to match or outperform the performances of BN with large-batch training.

Subfields: Biomedical Engineering, Hardware and Architecture, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Hardware and Architecture

In Optimization & Training

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, positive): “Layernorm (LN) is applied before every block, and residual connections after every block (Wang et al., 2019; Baevski & Auli, 2019).”
Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “We investigated Group Normalization (GN) [60] and Weight Standardization (WS) [43] as alternatives to BN.”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, methodology): “To avoid mismatch in batch statistics between the two data sources, in the co-training experiments we replace Batch-Norm with GroupNorm [62] and also standardize [45] the weights of the convolutions.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “In addition, Brock et al. (2021) prevent the emergence of a mean-shift in the hidden activations by introducing Scaled Weight Standardization (a minor modification of Weight Standardization (Huang et al., 2017; Qiao et al., 2019)).”
Getting to 99% Accuracy in Interactive Segmentation (2020, methodology): “The Image Stream architecture is based on ResNet-50 with Group Normalisation and Weight Standardisation [32].”
MMDetection: Open MMLab Detection Toolbox and Benchmark (2019, background): “• Weight Standardization [26]: standardizing the weights in the convolutional layers for micro-batch training, proposed in 2019.”
iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, methodology): “The encoder of the policy network consists of 3 parts: (a) a 9-layer ResNet with Weight Standardization [66] and GroupNorm, followed by a 3-layer Conv1D block to encode depth maps; (b) a 2-layer MLP to encode sensor observations; (c) a 2-layer fusion MLP to encode the concatenation of depth maps…”
Scaling Wide Residual Networks for Panoptic Segmentation (2020, background): “…of image recognition tasks, including image classiﬁcation [46, 77], object detection [22, 72], and dense prediction [61, 5], thanks to the recent advances in learning algorithms [23] ( e.g ., better optimizer [42], normalization techniques [40, 90, 70], and scalable training systems [1, 24, 66]).”
Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, methodology): “To resolve this, we propose Scaled Weight Standardization, a minor modiﬁcation of the recently proposed Weight Standardization (Qiao et al., 2019; Huang et al., 2017b), which prevents the growth in the mean signal, leading to a substantial boost in performance.”
Large Scale Learning of General Visual Representations for Transfer (2019, methodology): “Upstream Pre-Training For all of our models, we use a vanilla ResNet-v2 architecture [14], except that we replace all Batch Normalization [19] layers with Group Normalization [49] and use Weight Standardization [36] in all convolutional layers.”
Bootstrapped Representation Learning on Graphs (2021, methodology): “This aligns with recent observations that carefully chosen auxiliary losses are often paramount for stability of GAT models (Kim & Oh, 2021; Wang et al., 2019).”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “The idea of combining normalizing weights and activations to improve performance has been widely studied [24], [52], [141], [171].”
Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection (2019, methodology): “Apart from the above improvements, we ﬁnd that SENet [11], Weight Standardization [22] can also help in the detection task when used properly.”
Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift (2020, methodology): “…al., 2017] mixes data from the source and target domains during training time before passing them through the batch norm layers, whereas TransNorm [Wang et al., 2019] uses source and target domain batch norm statistics during training in an end-to-end fashion to improve transferability across…”
Pipelined Backpropagation at Scale: Training Large Models without Batches (2020, methodology): “The use of other small batch size alternatives to BN such as Online Normalization (Chiley et al., 2019), Weight Standardization (Qiao et al., 2019) or Filter Response Normalization (Singh & Kr-ishnan, 2019) may boost delay tolerance.”
SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection (2019, methodology): “Instead of using BN, which is not suitable for small batch size training, we adopt Weight Standardization (WS) (Qiao et al. 2019) for the weights in the convolution layers to further smooth the loss landscape.”
SPeCiaL: Self-Supervised Pretraining for Continual Learning (2021, methodology): “We use Weight Stan-dardized [ Qiao et al. , 2019 ] convolutions with Group Nor-malization [ Wu and He, 2018 ] , a normalization approach with does not use batch-level statistics and has shown strong performance in small batch settings.”
Deep Interactive Thin Object Selection (2021, methodology): “In order to accommodate the batch size of 1, we follow [10] by replacing all the batch normalization (BN) layers with group normalization (GN) [47] with weight standardization [38].”
Context-Gated Convolution (2019, methodology): “We conjecture that this is because the adaptiveness to global context improves the model’s generalization ability and the gating mechanism reduces the norm of gradients back-propagated to the convolution kernels, which leads to a smaller Lipschitz constant and thus better training stability [46,42].”