Adaptive Batch Normalization for practical domain adaptation

By Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, Jiaying Liu. Pattern Recognition, 2018

629 citations recorded by the source corpus.

Abstract Deep neural networks (DNN) have shown unprecedented success in various computer vision applications such as image classification and object detection. However, it is still a common annoyance during the training phase, that one has to prepare at least thousands of labeled images to fine-tune a network to a specific domain. Recent study (Tommasi et al., 2015) shows that a DNN has strong dependency towards the training dataset, and the learned features cannot be easily transferred to a different but relevant task without fine-tuning. In this paper, we propose a simple yet powerful remedy, called Adaptive Batch Normalization (AdaBN) to increase the generalization ability of a DNN. By modulating the statistics from the source domain to the target domain in all Batch Normalization layers across the network, our approach achieves deep adaptation effect for domain adaptation tasks. In contrary to other deep learning domain adaptation methods, our method does not require additional components, and is parameter-free. It archives state-of-the-art performance despite its surprising simplicity. Furthermore, we demonstrate that our method is complementary with other existing methods. Combining AdaBN with existing domain adaptation treatments may further improve model performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9f9c8cc77dafd3b1f2b615e97c9322f4346b5456

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multi-Modal Domain Adaptation for Fine-Grained Action Recognition (2019, methodology): “D2→ D1 D3→ D1 D1→ D2 D3→ D2 D1→ D3 D2→ D3 Mean MM Source-only 42.5 44.3 42.0 56.3 41.2 46.5 45.5 AdaBN [29] 44.6 47.8 47.0 54.7 40.3 48.8 47.2 MMD [32] 43.1 48.3 46.6 55.2 39.2 48.5 46.8 MCD [45] 42.1 47.9 46.5 52.7 43.5 51.0 47.3 The self-supervision hyper-parameter, λc = 5 was chosen by observing ”
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization (2021, methodology): “Li et al. (2018) took advantage of BN for tackling the domain adaptation problem.”
Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (2019, methodology): “AdaBN [7] proposes a post-processing method to re-estimate batch normalization statistics using target samples.”
A Decade Survey of Transfer Learning (2010–2020) (2020, methodology): “In addition, there are more models [3], [27], [28] that have used adaptation layer.”
AdaGraph: Unifying Predictive and Continuous Domain Adaptation Through Graphs (2019, methodology): “Following recent advances in DA [3, 20, 23], we propose to learn a set of domain-specific models by considering a common backbone network with domain-specific alignment layers embedded into it.”
Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, methodology): “We validate or method in the DA context against four existing unsupervised domain adaptation approaches: (a) AdaBN [32]: Batch Normalization layers are updated with target domain statistics; (b) MMD [36]: it minimizes separate discrepancy measures applied to single modalities; (c) Adversarial Only [”
Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation (2019, background): “Adaptive batch normalization (AdaBN) [33] has also been proposed to modulate the statistics in all batch normalization layers across the network between domains.”
Interactive Text-to-Speech via Semi-supervised Style Transfer Learning (2020, background): “In addition, AdaBN [23] is implemented in this model to boost domain adaptation performance between the internal and external datasets.”
Deep Learning for EMG-based Human-Machine Interaction: A Review (2021, positive): “[152] propose a framework that decodes hand movements robustly using supportive CNN classifiers.”
Temporal Attentive Alignment for Large-Scale Video Domain Adaptation (2019, methodology): “As mentioned in the experimental setup, we compare our proposed TA(3)N with other approaches by extending several state-of-the-art image-based DA methods [12, 26, 22, 38] for video DA with our TemPooling and TemRelation architectures, which are shown as follows:”
Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives (2022, methodology): “Zellinger et al. (2017) compare their CMD methods with other discrepancy-based methods, e.g., DDC (Tzeng et al., 2014), deep CROAL (Sun and Saenko, 2016), DLID (Chopra et al., 2013), AdaBN (Li et al., 2016)) and adversarial DANN (Ganin and Lempitsky, 2015).”
A Survey of Unsupervised Deep Domain Adaptation (2018, background): “In AdaBN, the target data are not used to learn the network weights but only for adjusting the batch norm statistics.”