Unsupervised Domain Adaptation by Backpropagation

By Yaroslav Ganin, Victor S. Lempitsky. 2014

7,030 citations recorded by the source corpus.

Top-performing deep architectures are trained on massive amounts of labeled data. In the absence of labeled data for a certain task, domain adaptation often provides an attractive option given that labeled data of similar nature but from a different domain (e.g. synthetic images) are available. Here, we propose a new approach to domain adaptation in deep architectures that can be trained on large amount of labeled data from the source domain and large amount of unlabeled data from the target domain (no labeled target-domain data is necessary). As the training progresses, the approach promotes the emergence of "deep" features that are (i) discriminative for the main learning task on the source domain and (ii) invariant with respect to the shift between the domains. We show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a simple new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation. Overall, the approach can be implemented with little effort using any of the deep-learning packages. The method performs very well in a series of image classification experiments, achieving adaptation effect in the presence of big domain shifts and outperforming previous state-of-the-art on Office datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

An Overview of Multi-Task Learning in Deep Neural Networks (2017, methodology): “This setup has found recent success in domain adaptation [Ganin and Lempitsky, 2015].”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “…et al., 2018b; 2019; Peng et al., 2019; Kamath et al., 2020), as well as from synthetic data to real data (including sim2real) (Wang et al., 2015; Ganin & Lempitsky, 2015; Richter et al., 2016; Peng et al., 2018; Hellendoorn et al., 2019; Xu et al., 2020; Bellemare et al., 2020), as synthetic…”
DensePose: Dense Human Pose Estimation in the Wild (2018, methodology): “Based on these results we rely exclusively on the DensePose dataset for training in the remaining experiments, even though domain adaptation [9] could be used in the future to exploit synthetic sources of supervision.”
Refining Language Models with Compositional Explanations (2021, background): “In another relevant thread, unsupervised domain adaptation (UDA) looks to adapt a trained model to a new domain using unlabeled data from the target domain, by updating feature representation to minimize the distribution divergence between domains [12, 11], or updating models to match the distributi”
Generative adversarial networks (2014, neutral): “3 GAN-like models called domain-adversarial networks can be used for domain adaptation.(12) GANs can be used for a variety of interactive digital media effects where the end goal is to produce compelling imagery.”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “Ganin and Lempitsky [6] use synthetic data in a domain adaptation setting where the learned features are invariant to the domain shift between synthetic and real images.”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, methodology): “Ganin and Lempitsky (Ganin and Lempitsky, 2014) employed a similar idea to use two models for domain adaptation.”
Domain Adaptation for Traffic Density Estimation (2021, background): “Traditional UDA approaches have been developed to address the problem of image classification, and they try to align features across the two domains ((Ganin and Lempitsky, 2015), (Tzeng et al., 2017)).”
Domain-Adversarial Training of Neural Networks (2015, background): “This version extends Ganin and Lempitsky (2015) very considerably by incorporating the report Ajakan et al. (2014) (presented as part of the Second Workshop on Transfer and Multi-Task Learning ), which brings in new terminology, in-depth theoretical analysis and justiﬁcation of the approach,…”
Adversarial Discriminative Domain Adaptation (2017, methodology): “The gradient reversal algorithm (ReverseGrad) proposed in [11] also treats domain invariance as a binary classiﬁcation problem, but directly maximizes the loss of the domain classiﬁer by reversing its gradients.”
A Theory of Multiple-Source Adaptation with Limited Target Labeled Data (2020, methodology): “We first evaluated our algorithm on a standard digits MSA dataset composed of four domains: MNIST (LeCun and Cortes, 2010), MNIST-M (Ganin and Lempitsky, 2015), SVHN (Netzer et al., 2011), and SynthDigits (Ganin and Lempitsky, 2015), by treating one of MNIST, MNIST-M, SVHN, or SynthDigits as the tar”
Agnostic Federated Learning (2019, neutral): “…and Mohri, 2014; Cortes, Mohri, and Mu˜noz Medina, 2015), either through unsupervised adaptation techniques (Gong et al., 2012; Long et al., 2015; Ganin and Lempitsky, 2015; Tzeng et al., 2015), or via lightly supervised ones (some amount of labeled data from the target domain) (Saenko et al.,…”
Algorithms and Theory for Multiple-Source Adaptation (2018, background): “…of transfer from a single source to a known target domain, either through unsupervised adaptation techniques [Gong et al., 2012, Long et al., 2015, Ganin and Lempitsky, 2015, Tzeng et al., 2015], or via lightly supervised ones (some amount of labeled data from the target domain) [Saenko et al.,…”
Learning Anonymized Representations with Adversarial Neural Networks (2018, methodology): “For this purpose, we introduce a novel training objective and framework inspired by Generative Adversarial Networks (GAN) by [Goodfellow et al., 2014] and by the domain adaptation framework of [Ganin and Lempitsky, 2015].”
Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning (2016, neutral): “2014; Gallagher, Tang, and Tu 2015) and/or datasets (Ganin and Lempitsky 2015).”
FRAGE: Frequency-Agnostic Word Representation (2018, background): “Our work is also weakly related to adversarial domain adaptation which attempts to mitigate the negative effects of domain shift between training and testing [9, 36].”