Differentiable Learning-to-Normalize via Switchable Normalization

By Ping Luo, Jiamin Ren, Zhanglin Peng. 2018

193 citations recorded by the source corpus.

We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization layers of a deep neural network. SN employs three distinct scopes to compute statistics (means and variances) including a channel, a layer, and a minibatch. SN switches between them by learning their importance weights in an end-to-end manner. It has several good properties. First, it adapts to various network architectures and tasks (see Fig.1). Second, it is robust to a wide range of batch sizes, maintaining high performance even when small minibatch is presented (e.g. 2 images/GPU). Third, SN does not have sensitive hyper-parameter, unlike group normalization that searches the number of groups as a hyper-parameter. Without bells and whistles, SN outperforms its counterparts on various challenging benchmarks, such as ImageNet, COCO, CityScapes, ADE20K, and Kinetics. Analyses of SN are also presented. We hope SN will help ease the usage and understand the normalization techniques in deep learning. The code of SN has been made available in this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Weight Standardization (2019, methodology): “This is to compare with the previous work that focus on micro-batch training problem, e.g. Switchable Normalization [26] and Group Normalization [42].”
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, methodology): “We consider two continual learning strategies: (i) Experience Replay (ER) (Chaudhry et al., 2019b); and (ii) Dark Experience Replay++ (DER++) (Buzzega et al., 2020 (Wu & He, 2018); and (v) SN (Luo et al., 2018).”
Four Things Everyone Should Know to Improve Batch Normalization (2019, background): “More recently, Switchable Normalization [19] aims to learn a more effective normalizer by calculating µi and σ2i as learned weighted combinations of the statistics computed from other normalization methods.”
Evolving Normalization-Activation Layers (2020, methodology): “For example, we avoid leveraging handcrafted normalization schemes (Luo et al., 2019) which are potentially suboptimal; we search for tensor-to-tensor transformations rather than scalar-to-scalar transformations (Ramachandran et al., 2018).”
Style Normalization and Restitution for Generalizable Person Re-Identification (2020, methodology): “SN [29] can be regarded as an adaptive ensemble version of normalization techniques of IN, BN, and LN (Layer Normalization) [2].”
Towards Understanding Regularization in Batch Normalization (2018, background): “Understanding the characteristics of these normalizers should be the first step to analyze some recent best practices such as whitening (Luo, 2017b;a), switchable normalization (Luo et al., 2019; 2018; Shao et al., 2019), and switchable whitening (Pan et al.”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, background): “In order to address such an issue, a significant number of standardization approaches are proposed [3, 48, 34, 30, 18, 27, 45, 24, 7].”
SSN: Learning Sparse Switchable Normalization via SparsestMax (2019, methodology): “For fair comparison with SN [14], we also adopted DeepLab [3] with ResNet50 as the backbone network, where output stride=8 and the last two blocks in the original ResNet contains atrous convolution layers with rate=2 and rate=4, respectively.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “results on several visual benchmarks [118].”
Learning to Optimize Domain Specific Normalization for Domain Generalization (2019, methodology): “Various normalization techniques [11, 27, 1, 19, 24, 29, 22, 16, 25, 5] have been studied actively in recent years.”
RelGAN: Multi-Domain Image-to-Image Translation via Relative Attributes (2019, methodology): “We use switchable normalization [18] in the generator.”
CNN-Based Learnable Gammatone Filterbank and Equal-Loudness Normalization for Environmental Sound Classification (2020, background): “The proposed normalization referred to as EN learns weighted statistics ofR different time-frequency regions to find a learned trade-off between energy equalization and structure preservation for ESC, which is similarly considered in [29], [30].”
Domain Generalization: A Survey (2021, background): “Following [234], a general formulation for different normalization layers can be written as γ x−μ σ + β, where μ and σ denote mean and variance respectively; γ and β are learnable scaling and shift parameters respectively.”
Low-Memory Neural Network Training: A Technical Report (2019, background): “, 2016], batch renormalization [Ioffe, 2017], group normalization [Wu and He, 2018], activation normalization [Kingma and Dhariwal, 2018], switchable normalization [Luo et al., 2019], and fixup initialization [Zhang et al.”
Positional Normalization (2019, methodology): “Normalization methods for neural networks can be roughly categorized into two regimes: normalization of weights [50, 55, 59, 73] and normalization of activations [29, 31, 37, 41, 47, 49, 61, 68, 74].”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “Switchable Normalization (SN) [215] introduces to tackle the different normalization techniques used for different convolutional layers within a CNN.”