Learning Transferable Features with Deep Adaptation Networks

By Mingsheng Long, Yue Cao, Jianmin Wang, Michael I. Jordan. 2015

6,016 citations recorded by the source corpus.

Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks for domain adaptation. However, as deep features eventually transition from general to specific along the network, the feature transferability drops significantly in higher layers with increasing domain discrepancy. Hence, it is important to formally reduce the dataset bias and enhance the transferability in task-specific layers. In this paper, we propose a new Deep Adaptation Network (DAN) architecture, which generalizes deep convolutional neural network to the domain adaptation scenario. In DAN, hidden representations of all task-specific layers are embedded in a reproducing kernel Hilbert space where the mean embeddings of different domain distributions can be explicitly matched. The domain discrepancy is further reduced using an optimal multikernel selection method for mean embedding matching. DAN can learn transferable features with statistical guarantees, and can scale linearly by unbiased estimate of kernel embedding. Extensive empirical evidence shows that the proposed architecture yields state-of-the-art image classification error rates on standard domain adaptation benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “For this reason, most work in CV focuses on transferring the last layers of the model (Long et al., 2015b).”
Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “omputer vision community has devoted signiﬁcant study to the problem of adapting vision-based models trained in a source domain to a previously unseen target domain (see, e.g., [9], [14], [15], [19], [23], [25], [51]). A variety of approaches have been proposed, including re-training the model in th”
Transfer Learning in Natural Language Processing (2019, background): “Optimization schedules for the adaptation phase can involve fine-tuning a varying portion of the pre-trained model (Long et al., 2015; Felbo et al., 2017; Howard and Ruder, 2018) with specifically designed regularization (Wiese et al.”
Fine-tuned Language Models for Text Classification (2018, methodology): “For this reason, most work in CV focuses on transferring the last layers of the model (Long et al., 2015b).”
One-Shot Imitation Learning (2017, background): “Success stories include domain adaptation in computer vision [64, 34, 28, 4, 15, 24, 33, 59, 14] and control [60, 45, 46, 20, 54].”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “Another line of research on domain adaptation attempt to match the distance between different domain distributions in a low dimensional space (Long et al., 2015; Baktashmotlagh et al., 2013).”
Domain-Adversarial Training of Neural Networks (2015, methodology): “Recently, several papers with significantly improved re-identification performance (Zhang and Saligrama, 2014; Zhao et al., 2014; Paisitkriangkrai et al., 2015) have been presented, with Ma et al. (2015) reporting good results in cross-data-set evaluation scenario.”
Learning without Forgetting (2016, background): “[22] matches the RKHS embedding of the deep representation of both source and target tasks to reduce domain bias.”
Adversarial Discriminative Domain Adaptation (2017, methodology): “This is generally achieved by optimizing the representation to minimize some measure of domain shift such as maximum mean discrepancy [5, 6] or correlation distances [7, 8].”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “In domain adaptation, for instance, the MMD has also been used to reduce the dataset bias and thus improve transferability of the features across different domains (Long et al. 2015; 2016).”
Agnostic Federated Learning (2019, background): “(Gong et al., 2012; Long et al., 2015; Ganin and Lempitsky, 2015; Tzeng et al., 2015), or via lightly supervised ones (some amount of labeled data from the target domain) (Saenko et al.”
Algorithms and Theory for Multiple-Source Adaptation (2018, background): “The problem of transfer from a single source to a known target domain (Ben-David, Blitzer, Crammer, and Pereira, 2006; Mansour, Mohri, and Rostamizadeh, 2009b; Cortes and Mohri, 2014; Cortes, Mohri, and Muñoz Medina, 2015), either through unsupervised adaptation techniques (Gong et al., 2012; Long e”
Knowledge Flow: Improve Upon Your Teachers (2019, background): “…2016), learning without forgetting (Li & Hoiem, 2016), growing a brain (Wang et al., 2017), policy distillation (Rusu et al., 2016a), domain adaptation (Pan & Yang, 2010; Long et al., 2015; Tzeng et al., 2015), knowledge distillation (Hin-ton et al., 2015) or lifelong learning (Chen & Liu, 2016).”
The Variational Fair Autoencoder (2015, methodology): “This is done with either MMD (Long & Wang, 2015; Tzeng et al., 2014) or adversarial (Ganin et al., 2015) penalties at the hidden layers of the network.”
Domain Adaptation with Conditional Transferable Components (2016, methodology): “Other methods learn a low dimensional projection by explicitly minimizing the discrepancy between the distributions of projected features on source and target domains (Pan et al., 2011; Long et al., 2014; 2015; Baktashmotlagh et al., 2013; Si et al., 2010; 2011; Muan-det et al., 2013).”
Learning to cluster in order to Transfer across domains and tasks (2017, background): “…there has recently been a large body of work dealing with domain shift between image datasets by minimizing domain discrepancy (Tzeng et al., 2014; Long et al., 2015b; Ganin et al., 2016; Sun et al., 2016; Long et al., 2016; Sener et al., 2016; Carlucci et al., 2017; Long et al., 2017; Zellinger…”
Few-Shot Adversarial Domain Adaptation (2017, background): “LB [60] [34] [15] [58] [27] [38] FADA A →W 61.”