Beyond Sharing Weights for Deep Domain Adaptation

By Artem Rozantsev, Mathieu Salzmann, Pascal Fua. 2016

473 citations recorded by the source corpus.

The performance of a classifier trained on data coming from a specific domain typically degrades when applied to a related but different one. While annotating many samples from the new domain would address this issue, it is often too expensive or impractical. Domain Adaptation has therefore emerged as a solution to this problem; It leverages annotated data from a source domain, in which it is abundant, to train a classifier to operate in a target domain, in which it is either sparse or even lacking altogether. In this context, the recent trend consists of learning deep architectures whose weights are shared for both domains, which essentially amounts to learning domain invariant features. Here, we show that it is more effective to explicitly model the shift from one domain to the other. To this end, we introduce a two-stream architecture, where one operates in the source domain and the other in the target domain. In contrast to other approaches, the weights in corresponding layers are related but not shared. We demonstrate that this both yields higher accuracy than state-of-the-art methods on several object recognition and detection tasks and consistently outperforms networks with shared weights in both supervised and unsupervised settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Few-Shot Adversarial Domain Adaptation (2017, methodology): “The first two tasks includeM→ U , U →M, and followed the experimental setting in [12, 45, 33, 59, 49], which involves randomly selecting 2000 images from MNIST and 1800 images from USPS.”
Progress & Compress: A scalable framework for continual learning (2018, background): “, 2016) which introduces a gating mechanism between columns and (Rozantsev et al., 2016), who formulate an alternative regularisation objective to keep weights of columns tied.”
Unified Deep Supervised Domain Adaptation and Generalization (2017, methodology): “We considered two cross-domain tasks,M→ U and U →M, and followed the experimental setting in [20, 50], which involves randomly selecting 2000 images from MNIST and 1800 images from USPS.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “[183] consider the MMD between the weights of the source respectively target models of different layers, where an extra regularizer term ensures that the weights in the two models remains linearly related.”
Explicit Inductive Bias for Transfer Learning with Convolutional Networks (2018, methodology): “Rozantsev et al. (2016) introduced a parameter regularization for keeping the similarity between the pre-trained model and the ﬁne-tuned model.”
Label Efficient Learning of Transferable Representations acrosss Domains and Tasks (2017, result): “Weight sharing of CNN parameters [58, 22, 21, 3] and minimizing the distribution discrepancy of network activations [51, 65, 30] have also shown convincing results.”
Agnostic Domain Generalization (2018, background): “The idea is to introduce different measures of domain distribution shift at a single or multiple levels in the architecture [26, 41, 4, 5, 32] and then train the network to minimize these measures while also reducing a task-speciﬁc (e.g. classiﬁcation, detection) loss.”
A Survey of Unsupervised Deep Domain Adaptation (2018, methodology): “ ±0.3 73.4 ±0.8 71.8 ±0.6 MADA[183]a∗ 90.0 ±0.1 97.4 0.1 99.6 ±0.1 87.8 ±0.2 70.3 ±0.3 66.4 0.3 AutoDIAL[27]bc 84.2 97.9 99.9 82.3 64.6 64.2 CCN++[101]d∗ 78.2 97.4 98.6 73.5 62.8 60.6 Rozantsev et al.[199] 76.0 96.7 99.6 AdaBN[145]b 74.2 95.7 99.8 73.1 59.8 57.4 JAN-A[151]a∗ 86.0 ±0.4 96.7 ±0.3 99.7”
Incremental Adversarial Domain Adaptation for Continually Changing Environments (2017, background): “[17] extend in a similar direction and impose a penalty for deviations in the network parameters across domains.”
Domain-Adaptive Few-Shot Learning (2020, background): “al FSL [48, 49, 35, 21, 57, 52, 41] nor DA [54, 12, 3, 30, 2, 19, 62, 43, 50, 22] methods can be directly extended to DA-FSL, though they have drawn much attention.”
Multicenter Computer-Aided Diagnosis for Lymph Nodes Using Unsupervised Domain-Adaptation Networks Based on Cross-Domain Confounding Representations (2020, background): “Tzeng et al. [9] and Rozantsev et al. [10] use MMD to measure the distance between two feature spaces and to map the source and the target spaces to a uniﬁed feature space by minimizing this distance.”
Adversarial Discriminative Domain Adaptation (2017, background): “Rozantsev et al. [17] showed that partially shared weights can lead to effective adaptation in both supervised and unsupervised settings.”
Model Adaptation: Unsupervised Domain Adaptation Without Source Data (2020, background): “Inspired by [46, 57] which attempt to learn two separate but related prediction models for the source and target domains, we propose a weight regularization term ℓ wReg to prevent the parameters of the prediction model C to drift far away from those of the pre-trained model learnt in the source…”
A Decade Survey of Transfer Learning (2010–2020) (2020, methodology): “Generally, there are two types of feature-based transductive learning methods: data-centered methods [16]–[19], [23] and subspace-centered methods [21], [22], [24]–[26].”
Deep Visual Domain Adaptation: A Survey (2018, methodology): “[47] considered that the weights in correspondng layers are not shared but related by a weight regularizer r w ( · ) o account for the differences between the two domains (Fig.”