Double Forward Propagation for Memorized Batch Normalization
By Yong Guo, Qingyao Wu, Chaorui Deng, Mingkui Tan. 2018
29 citations recorded by the source corpus.
Batch Normalization (BN) has been a standard component in designing deep neural networks (DNNs). Although the standard BN can significantly accelerate the training of DNNs and improve the generalization performance, it has several underlying limitations which may hamper the performance in both training and inference. In the training stage, BN relies on estimating the mean and variance of data using a single mini-batch. Consequently, BN can be unstable when the batch size is very small or the data is poorly sampled. In the inference stage, BN often uses the so called moving mean and moving variance instead of batch statistics, i.e., the training and inference rules in BN are not consistent. Regarding these issues, we propose a memorized batch normalization (MBN), which considers multiple recent batches to obtain more accurate and robust statistics. Note that after the SGD update for each batch, the model parameters will change, and the features will change accordingly, leading to the Distribution Shift before and after the update for the considered batch. To alleviate this issue, we present a simple Double-Forward scheme in MBN which can further improve the performance. Compared to related methods, the proposed MBN exhibits consistent behaviors in both training and inference. Empirical results show that the MBN based models trained with the Double-Forward scheme greatly reduce the sensitivity of data and significantly improve the generalization performance.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Discrimination-aware Channel Pruning for Deep Neural Networks (2018, background): “, image classification [19, 38, 7], face recognition [34, 39], object detection [32, 33], image captioning [44, 47] and video analysis [35, 45].”
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “[129] introduced memorized BN, which considers data information from multiple recent batches (or all batches in an extreme case) to produce more accurate and stable statistics.”
- NAT: Neural Architecture Transformer for Accurate and Compact Architectures (2019, background): “Deep neural networks (DNNs) [25] have been producing state-of-the-art results in many challenging tasks including image classification [12, 23, 43, 60, 61, 18, 11, 56], face recognition [39, 44, 59], brain signal processing [34, 35], video analysis [48, 53, 52] and many other areas [58, 57, 24, 3,…”
- Attend and Imagine: Multi-Label Image Classification With Visual Attention and Recurrent Neural Networks (2019, background): “D URING the past few years, thanks to the development of learning theory [1]–[3] and the availability of large-scale datasets like ImageNet [4], significant progress has been made in the task of single-label image classification [5]–[10], where each image is associated with one label only.”
- Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution (2020, methodology): “Deep neural networks (DNNs) have been the workhorse of many real-world applications, including image classification [18, 14, 9, 15, 27, 13], video understanding [46, 45,”
- Discrimination-Aware Network Pruning for Deep Model Compression (2020, methodology): “Moreover, to accelerate the convergence, we apply batch normalization [15], [29] and ReLU [54] before performing the average”
- Four Things Everyone Should Know to Improve Batch Normalization (2019, background): “Guo et al. (2018) tackle the small batch setting by aggregating normalization statistics over multiple forward passes.”