Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes

By Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H Sinz, Richard S. Zemel. 2016

86 citations recorded by the source corpus.

Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural networks. On the other hand, layer normalization normalizes the activations across all activities within a layer. This was shown to work well in the recurrent setting. In this paper we propose a unified view of normalization techniques, as forms of divisive normalization, which includes layer and batch normalization as special cases. Our second contribution is the finding that a small modification to these normalization schemes, in conjunction with a sparse regularizer on the activations, leads to significant benefits over standard normalization techniques. We demonstrate the effectiveness of our unified divisive normalization framework in the context of convolutional neural nets and recurrent neural networks, showing improvements over baselines in image classification, language modeling as well as super-resolution.

Subfields: Artificial Intelligence, Geophysics, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Geophysics

Kaiming He

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Weight Standardization (2019, methodology): “To explore other dimensions [2, 33], Layer Normalization [3] normalizes data on the channel dimension, Instance Normalization [40] performs BN for each sample individually, and Group Nor-malization [42] ﬁnds a better middle point between Layer Normalization and Instance Normalization.”
Group Normalization (2018, methodology): “Several normalization methods [17,18,19,33,34] have been proposed to avoid exploiting the batch dimension.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “Recently, several alternative normalization schemes have been proposed to extend BN’s effectiveness to recurrent architectures [35, 2, 47, 8, 29, 44].”
Generalisation in humans and deep neural networks (2018, background): “These normalisation processes seem to be crucial for many aspects of robust animal and human vision [46], are predictive for human vision data [21, 64] and have proven useful in the context of computer vision [65, 66].”
Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks (2017, background): “For example, pre-training [6, 7], special network structure [20, 24, 4], ReLU activation [16, 14], noise injecting [26, 23], normalization [9, 18, 3, 2, 17].”
Decorrelated Batch Normalization (2018, unknown): “under the Division Normalization framework [39].”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, background): “In order to address such an issue, a significant number of standardization approaches are proposed [3, 48, 34, 30, 18, 27, 45, 24, 7].”
Differentiable Learning-to-Normalize via Switchable Normalization (2018, background): “We would also like to acknowledge the contributions of previous work that explored spatial region (Ren et al., 2016) and conditional normalization (Perez et al.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “Divisive normalization (DN) [76] generalizes the neighborhood partition of these local normalization methods as the choices of the summation and suppression ﬁelds [76], [77].”
EvalNorm: Estimating Batch Normalization Statistics for Evaluation (2019, background): “[18] provide a unifying view of the different normalization approaches by characterizing them as the same transformation but along different dimensions (layers, samples, filters, etc.”
Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification (2017, background): “This group includes Layer Normalization (Ba et al. (2016)) which uses channel dimension instead of a batch di-mension to perform normalization, Instance Normalization (Ulyanov et al. (2016)) which only nor-malizes over the spatial dimensions of the output and Divisive Normalization (Ren et al. (2016”