EvalNorm: Estimating Batch Normalization Statistics for Evaluation

By Saurabh Singh, Abhinav Shrivastava. 2019

57 citations recorded by the source corpus.

Batch normalization (BN) has been very effective for deep learning and is widely used. However, when training with small minibatches, models using BN exhibit a significant degradation in performance. In this paper we study this peculiar behavior of BN to gain a better understanding of the problem, and identify a cause. We propose `EvalNorm' to address the issue by estimating corrected normalization statistics to use for BN during evaluation. EvalNorm supports online estimation of the corrected statistics while the model is being trained, and does not affect the training scheme of the model. As a result, EvalNorm can also be used with existing pre-trained models allowing them to benefit from our method. EvalNorm yields large gains for models trained with smaller batches. Our experiments show that EvalNorm performs 6.18% (absolute) better than vanilla BN for a batchsize of 2 on ImageNet validation set and from 1.5 to 7.0 points (absolute) gain on the COCO object detection benchmark across a variety of setups.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Soham De
2. Samuel L. Smith
3. Ling Shao
4. Andrew Brock
5. Fan Zhu
6. Quoc V. Le
7. Karen Simonyan
8. Li Liu

In Computer Vision and Pattern Recognition

1. Ling Shao
2. Soham De
3. Luc Van Gool
4. Samuel L. Smith
5. Dacheng Tao
6. Andrew Brock
7. Li Liu
8. Ming-Hsuan Yang

In Radiology, Nuclear Medicine and Imaging

1. Ling Shao
2. Jie Qin
3. Fan Zhu
4. Li Liu

In Model Architectures

1. Soham De
2. Ling Shao
3. Andrew Brock
4. Samuel L. Smith
5. Lei Huang
6. Li Liu
7. Fan Zhu
8. Yi Zhou

In Optimization & Training

1. Soham De
2. Andrew Brock
3. Samuel L. Smith
4. Karen Simonyan
5. Ling Shao
6. Jie Qin
7. Fan Zhu
8. Yi Zhou

Citations in the corpus

• High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Second, it introduces a discrepancy between the behaviour of the model during training and at inference time (Summers & Dinneen, 2019; Singh & Shrivastava, 2019), introducing hidden hyper-parameters that have to be tuned.”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “Singh et al . analyzed how a small batch size hampers the estimation accuracies of BN when using running averages, and proposed EvalNorm [123], which optimizes the sample weight during inference to ensure that the activations produced by normalization are similar to those provided during training.”