Bridging Theory and Algorithm for Domain Adaptation

By Yuchen Zhang, Tianle Liu, Mingsheng Long, Michael I. Jordan. 2019

844 citations recorded by the source corpus.

This paper addresses the problem of unsupervised domain adaption from theoretical and algorithmic perspectives. Existing domain adaptation theories naturally imply minimax optimization algorithms, which connect well with the domain adaptation methods based on adversarial learning. However, several disconnections still exist and form the gap between theory and algorithm. We extend previous theories (Mansour et al., 2009c; Ben-David et al., 2010) to multiclass classification in domain adaptation, where classifiers based on the scoring functions and margin loss are standard choices in algorithm design. We introduce Margin Disparity Discrepancy, a novel measurement with rigorous generalization bounds, tailored to the distribution comparison with the asymmetric margin loss, and to the minimax optimization for easier training. Our theory can be seamlessly transformed into an adversarial learning algorithm for domain adaptation, successfully bridging the gap between theory and algorithm. A series of empirical studies show that our algorithm achieves the state of the art accuracies on challenging domain adaptation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, positive): “In practice, the discrepancy distance Ld(·, ·) can be estimated as the upper bound based on the Rademacher complexity which is used in the domain adaptation theory as a measure of richness for a particular hypothesis space [30, 60].”
Unsupervised Domain Adaptation via Structurally Regularized Deep Clustering (2020, background): “Based on these assumptions, many of exiting works [16, 35, 42, 48, 53, 66] take the transferring strategy of learning aligned feature representations between the two domains, such that classifiers trained on source features can be readily applied to the target ones.”
Towards Discriminability and Diversity: Batch Nuclear-Norm Maximization Under Label Insufficient Situations (2020, neutral): “6 MDD [53] 54.”
On Localized Discrepancy for Domain Adaptation (2020, background): “First, the localized discrepancies can be significantly smaller than previous discrepancies Ben-David et al. (2010); Zhang et al. (2019).”
Source Data-Absent Unsupervised Domain Adaptation Through Hypothesis Transfer and Labeling Transfer (2020, methodology): “Following previous works [102], [104], we further adopt the ResNet-50 [118] backbone to validate the effectiveness of our methods.”
Minimum Class Confusion for Versatile Domain Adaptation (2019, methodology): “A rich line of DNN-based methods [44,21,23,24,49,42,8,43,30,22,47,53] have been proposed for Unsupervised DA (UDA), a closed-set scenario with one source domain and ar X iv :1 91 2.”
Margin-aware Adversarial Domain Adaptation with Optimal Transport (2020, background): “…2 ( h 1 − h 2 ) , the last two equations can be written: In particular, when we consider the class associated to a scoring function, i.e., y h := [ h 1 > h 2 ] corresponding in our case to y h = sgn ( h ) , we have: Hence, we have for all β > 0 , where l 0 ,β is denoted Φ β in (Zhang et al., 2019).”
Gradually Vanishing Bridge for Adversarial Domain Adaptation (2020, methodology): “We compare our methods with state-of-art domain adaptation methods: DAN [23], DANN [9], ADDA [40], GTA [36], CDAN [24], Simnet [32], MCD [35] and MDD [47].”
Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation (2021, neutral): “4 MDD [59] 7 - - - - - - - - - - - - 74.”
Continuously Indexed Domain Adaptation (2020, methodology): “ADDA and MDD merge data with different domain indices into one source and one target domains; DANN, CDANN, and CUA divide the continuous domain spectrum into several separate domains and perform adaptation between multiple source and target domains.”
Continuous Domain Adaptation with Variational Domain-Agnostic Feature Replay (2020, methodology): “…between the support data and query data, while most unsupervised domain adaptation algorithms (Long et al., 2015; Ganin et al., 2016; Tzeng et al., 2017; Hoffman et al., 2018; Saito et al., 2018; Zhang et al., 2019) resolve a single step of the domain drift without considering the stream data.”
Learning to Adapt to Evolving Domains (2020, background): “[6, 19, 35, 31, 20, 40] learn domain-invariant representations via domain adversarial training.”
Learning Bounds for Open-Set Learning (2021, neutral): “Deﬁnition 4 (Disparity Discrepancy (Zhang et al., 2019)) .”