Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise

By Senwei Liang, Zhongzhan Huang, Mingfu Liang, Haizhao Yang. 2019

66 citations recorded by the source corpus.

Batch Normalization (BN) (Ioffe and Szegedy 2015) normalizes the features of an input image via statistics of a batch of images and hence BN will bring the noise to the gradient of training loss. Previous works indicate that the noise is important for the optimization and generalization of deep neural networks, but too much noise will harm the performance of networks. In our paper, we offer a new point of view that the self-attention mechanism can help to regulate the noise by enhancing instance-specific information to obtain a better regularization effect. Therefore, we propose an attention-based BN called Instance Enhancement Batch Normalization (IEBN) that recalibrates the information of each channel by a simple linear transformation. IEBN has a good capacity of regulating the batch noise and stabilizing network training to improve generalization even in the presence of two kinds of noise attacks during training. Finally, IEBN outperforms BN with only a light parameter increment in image classification tasks under different network structures and benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhongzhan Huang
2. Senwei Liang
3. Wei He
4. Fan Zhu
5. Haizhao Yang
6. Mingfu Liang
7. Ling Shao
8. Li Liu

In Computer Vision and Pattern Recognition

1. Zhongzhan Huang
2. Senwei Liang
3. Ling Shao
4. Wei He
5. Haizhao Yang
6. Mingfu Liang
7. Li Liu
8. Fan Zhu

Citations in the corpus

• AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “On the other hand, in the non-continual learning scenario, i.e ., learning all the classes at once, we know that when the batch size is too small, the distribution of each batch may extremely vary and cause the shifting of the loss landscape, leading to lousy generalization after training [22, 26].”
• Efficient Attention Network: Accelerate Attention by Searching Where to Plug (2020, methodology): “Recently, many plug-and-play and straightforward self-attention modules that utilize the interior information of a network to enhance instance specificity [21] are proposed to boost the generalization of deep convolutional neural net-* Figure 1.”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “It is believed such an improvement is obtained from the stochasticity/noise introduced by normalization over batch data [8], [105], [205].”