Four Things Everyone Should Know to Improve Batch Normalization

By Cecilia Summers, Michael J. Dinneen. 2019

57 citations recorded by the source corpus.

A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend themselves to other enhancements. In this paper, we identify four improvements to the generic form of Batch Normalization and the circumstances under which they work, yielding performance gains across all batch sizes while requiring no additional computation during training. These contributions include proposing a method for reasoning about the current example in inference normalization statistics, fixing a training vs. inference discrepancy; recognizing and validating the powerful regularization effect of Ghost Batch Normalization for small and medium batch sizes; examining the effect of weight decay regularization on the scaling and shifting parameters gamma and beta; and identifying a new normalization algorithm for very small batch sizes by combining the strengths of Batch and Group Normalization. We validate our results empirically on six datasets: CIFAR-100, SVHN, Caltech-256, Oxford Flowers-102, CUB-2011, and ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Soham De
2. Samuel L. Smith
3. Ling Shao
4. Andrew Brock
5. Fan Zhu
6. Quoc V. Le
7. Karen Simonyan
8. Li Liu

In Computer Vision and Pattern Recognition

1. Ling Shao
2. Soham De
3. Luc Van Gool
4. Samuel L. Smith
5. Dacheng Tao
6. Andrew Brock
7. Li Liu
8. Ming-Hsuan Yang

In Optimization & Training

1. Soham De
2. Andrew Brock
3. Samuel L. Smith
4. Karen Simonyan
5. Ling Shao
6. Jie Qin
7. Fan Zhu
8. Yi Zhou

Citations in the corpus

• High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Second, it introduces a discrepancy between the behaviour of the model during training and at inference time (Summers & Dinneen, 2019; Singh & Shrivastava, 2019), introducing hidden hyper-parameters that have to be tuned.”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “Batch group normalization (BGN) [73] expands the grouping mechanism of GN from being over only channels to being over both channels and batch dimensions.”