Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification

By Igor Gitman, Boris Ginsburg. 2017

69 citations recorded by the source corpus.

Batch normalization (BN) has become a de facto standard for training deep convolutional networks. However, BN accounts for a significant fraction of training run-time and is difficult to accelerate, since it is a memory-bandwidth bounded operation. Such a drawback of BN motivates us to explore recently proposed weight normalization algorithms (WN algorithms), i.e. weight normalization, normalization propagation and weight normalization with translated ReLU. These algorithms don't slow-down training iterations and were experimentally shown to outperform BN on relatively small networks and datasets. However, it is not clear if these algorithms could replace BN in practical, large-scale applications. We answer this question by providing a detailed comparison of BN and WN algorithms using ResNet-50 network trained on ImageNet. We found that although WN achieves better training accuracy, the final test accuracy is significantly lower ($\approx 6\%$) than that of BN. This result demonstrates the surprising strength of the BN regularization effect which we were unable to compensate for using standard regularization techniques like dropout and weight decay. We also found that training of deep networks with WN algorithms is significantly less stable compared to BN, limiting their practical applications.

Subfields: Computer Vision and Pattern Recognition, Industrial and Manufacturing Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Industrial and Manufacturing Engineering

In Optimization & Training

Citations in the corpus

High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “First, it is a surprisingly expensive computational primitive, which incurs memory overhead (Rota Bulò et al., 2018), and significantly increases the time required to evaluate the gradient in some networks (Gitman & Ginsburg, 2017).”
A Mean Field Theory of Batch Normalization (2019, background): “Comparisons of performance between these different schemes have been challenging and inconclusive (Gitman & Ginsburg, 2017).”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “However, normalizing weights has some drawbacks when used in large-scale networks in practice: 1) It may not effectively improve the optimization efficiency when residual connections are introduced or when the nonlinearity does not satisfy the assumptions required for preserving distributions betwee”
Deep learning classifiers for hyperspectral imaging: A review (2019, background): “This, coupled with the mean-only batch normalization, allows the scale of neural activations to be approximately independent of the parameter v , as well as their mean (Gitman and Ginsburg, 2017).”
EEG-TCNet: An Accurate Temporal Convolutional Network for Embedded Motor-Imagery Brain–Machine Interfaces (2020, methodology): “• Batch normalization is used between convolutions instead of weight normalization as batch normalization has been shown to give higher accuracy than weight normalization on various large scale networks [31].”
A Simple Convergence Proof of Adam and Adagrad (2020, unknown): “(b) Average squared norm of the gradient of a small convolutional model Gitman and Ginsburg (2017) trained on CIFAR-10, with a random initialization.”
Norm matters: efficient and accurate normalization schemes in deep networks (2018, methodology): “Our implementation of WN here could not converge (similar convergence issues were reported by [11]).”
Towards Understanding Regularization in Batch Normalization (2018, background): “Gitman & Ginsburg (2017) experimentally compared BN and WN, and also confirmed the better generalization of BN.”
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization (2018, methodology): “Interest-ingly, weight normalization has been shown empirically to beneﬁt from similar acceleration properties as Batch Normalization (Gitman and Ginsburg, 2017; Salimans and Kingma, 2016).”