Revisiting Batch Normalization For Practical Domain Adaptation

By Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, Xiaodi Hou. 2016

695 citations recorded by the source corpus.

Deep neural networks (DNN) have shown unprecedented success in various computer vision applications such as image classification and object detection. However, it is still a common annoyance during the training phase, that one has to prepare at least thousands of labeled images to fine-tune a network to a specific domain. Recent study (Tommasi et al. 2015) shows that a DNN has strong dependency towards the training dataset, and the learned features cannot be easily transferred to a different but relevant task without fine-tuning. In this paper, we propose a simple yet powerful remedy, called Adaptive Batch Normalization (AdaBN) to increase the generalization ability of a DNN. By modulating the statistics in all Batch Normalization layers across the network, our approach achieves deep adaptation effect for domain adaptation tasks. In contrary to other deep learning domain adaptation methods, our method does not require additional components, and is parameter-free. It archives state-of-the-art performance despite its surprising simplicity. Furthermore, we demonstrate that our method is complementary with other existing methods. Combining AdaBN with existing domain adaptation treatments may further improve model performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “…the model in the target domain (e.g., [52]), adapting the weights of the model based on the statistics of the source and target domains (e.g., [22]), learning invariant features between domains (e.g., [47]), and learning a mapping from the target domain to the source domain (e.g., [43]).”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “[34] found that BN can alleviate domain shifts by recomputing popular statistics in the target domain.”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “Specifically, before computing the distance, we perform Instance Normalization [71] (without affine transformations) on the VGG features in order to remove the original feature mean and variance, which contains much domain-specific information [54, 76].”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, neutral): “Note that some recent work offers a possible justification for using test set statistics for batch normalization, suggesting that it may be helpful in mitigating distribution shift [68].”
Unsupervised Model Personalization While Preserving Privacy and Scalability: An Open Problem (2020, background): “Whereas BN obtains global batch statistics of the training data for inference, Adaptive BN (AdaBN) [16] introduces an unsupervised scheme gathering batch statistics of the target domain data instead.”
Slimmable Neural Networks (2018, methodology): “Conditional normalization is widely used in tasks including style transfer (Dumoulin et al., 2016; Li et al., 2017a; Huang & Belongie, 2017; Li et al., 2017b), image recognition (Li et al., 2016b; Yang et al., 2018) and many others (Perez et al., 2017b;a).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “It is also common to split the data to induce a shift that is more extreme than one might typically encounter, e.g., in PACS (Li et al., 2017), which includes classifying photos solely from cartoons and other stylized representations; DeepFashion Remixed (Hendrycks et al., 2020), which involves…”
Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning (2017, methodology): “…approaches work particularly well with (deep) neural networks, which produce outstanding results on some domain adaptation data sets (Ganin et al., 2016; Sun & Saenko, 2016; Li et al., 2016; Aljundi et al., 2015; Long et al., 2015; Li et al., 2015; Zhuang et al., 2015; Louizos et al., 2016).”
Image Generation From Small Datasets via Batch Statistics Adaptation (2019, background): “Adaptive batch normalization [15] performs domain adaptation for the segmentation task by replacing the statistics of the source domain with those of the target domain, achieving performance accuracy competitive with other deep-learning based methods.”
Batch Coherence-Driven Network for Part-Aware Person Re-Identification (2020, methodology): “[48] argued that domain knowledge can be reflected by the statistics in the BN layer.”
Double Forward Propagation for Memorized Batch Normalization (2018, methodology): “Normalization depending on single data batch can be inaccurate and unstable when the batch size is very small or the data is poorly sampled (Li et al. 2016).”
Surface EMG-Based Inter-Session Gesture Recognition Enhanced by Deep Domain Adaptation (2017, methodology): “Our multi-stream extension of AdaBN [55] improves inter-session accuracy, especially when when a large number of recording sessions are presented in the training set.”
A New Deep Learning Model for Fault Diagnosis with Good Anti-Noise and Domain Adaptation Ability on Raw Vibration Signals (2017, methodology): “In 2016, Li et al. proposed a simple method named Adaptive Batch Normalization (AdaBN) [30] to utilize BN to endow neural networks with good domain adaptation capacity.”
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization (2021, methodology): “Here, an effective approach in practice is utilizing Batch Normalization (BN) (Ioffe & Szegedy, 2015): recent work has proposed BN as a tool to mitigate domain shifts in domain adaptation tasks with promising results achieved (Li et al., 2016; Liu et al., 2020; Chang et al., 2019).”
Tent: Fully Test-Time Adaptation by Entropy Minimization (2021, background): “For batch normalization Li et al. (2017); Carlucci et al. (2017) separate source and target statistics during training.”
Deep Visual Domain Adaptation: A Survey (2018, methodology): “Zellinger et al. [134] evaluated their proposed CMD algorithm in comparison to other discrepancy-based methods (DDC, deep CROAL [109], DLID [16], AdaBN [69]) and the adversarial-based method DANN.”