Domain Adaptation: Learning Bounds and Algorithms

By Yishay Mansour, Mehryar Mohri, Afshin Rostamizadeh. 2009

876 citations recorded by the source corpus.

This paper addresses the general problem of domain adaptation which arises in a variety of applications where the distribution of the labeled sample available somewhat differs from that of the test data. Building on previous work by Ben-David et al. (2007), we introduce a novel distance between distributions, discrepancy distance, that is tailored to adaptation problems with arbitrary loss functions. We give Rademacher complexity bounds for estimating the discrepancy distance from finite samples for different loss functions. Using this distance, we derive new generalization bounds for domain adaptation for a wide family of loss functions. We also present a series of novel adaptation bounds for large classes of regularization-based algorithms, including support vector machines and kernel ridge regression based on the empirical discrepancy. This motivates our analysis of the problem of minimizing the empirical discrepancy for various loss functions for which we also give several algorithms. We report the results of preliminary experiments that demonstrate the benefits of our discrepancy minimization algorithms for domain adaptation.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “The computer vision community has devoted significant study to the problem of adapting vision-based models trained in a source domain to a previously unseen target domain (see, e.g., [9], [14], [15], [19], [23], [25], [51]).”
Learning Robust Global Representations by Penalizing Local Predictive Power (2019, background): “bounded divergences between source and target distributions) (Mansour et al., 2009; Hu et al., 2016).”
One-Shot Imitation Learning (2017, background): “Success stories include domain adaptation in computer vision [64, 34, 28, 4, 15, 24, 33, 59, 14] and control [60, 45, 46, 20, 54].”
Domain-Adversarial Training of Neural Networks (2015, background): “Several notions of distance have been proposed for domain adaptation (Ben-David et al., 2006, 2010; Mansour et al., 2009a,b; Germain et al., 2013).”
Learning Transferable Features with Deep Adaptation Networks (2015, methodology): “We provide an analysis of the expected target-domain risk of our approach, making use of the theory of domain transfer (Ben-David et al., 2007; 2010; Mansour et al., 2009) and the theory of kernel embedding of probability distributions (Sriperumbudur et al.”
Advances and Open Problems in Federated Learning (2019, background): “Can we guarantee that the model hFL learned via federated learning is at least as accurate as hk when used for client k? Can we quantify how much improvement can be expected via federated leaning? And can we develop personalization strategies with theoretical guarantees that at least match the perfo”
Multiple-Source Adaptation with Domain Classifiers (2020, methodology): “…information available to the learner, the number of source domains, and the amount of labeled and unlabeled data available from the target domain [Mansour et al., 2008, 2009a, Hoffman et al., 2018, Pan and Yang, 2010, Muandet et al., 2013, Xu et al., 2014, Hoffman et al., 2012, Gong et al.,…”
A Theory of Multiple-Source Adaptation with Limited Target Labeled Data (2020, background): “It was shown by Mansour et al. (2009c) (see also Cortes and Mohri (2011)) that the discrepancy is the appropriate divergence between distributions in adaptation.”
Three Approaches for Personalization with Applications to Federated Learning (2020, methodology): “To obtain better bounds, we use the notion of label-discrepancy between distributions (Mansour et al., 2009).”
Agnostic Federated Learning (2019, background): “…the problem of transfer learning or domain adaptation from a single source to a known target domain (Ben-David, Blitzer, Crammer, and Pereira, 2006; Mansour, Mohri, and Rostamizadeh, 2009b; Cortes and Mohri, 2014; Cortes, Mohri, and Mu˜noz Medina, 2015), either through unsupervised adaptation…”
Algorithms and Theory for Multiple-Source Adaptation (2018, methodology): “This is a more general setup than the deterministic scenario in [Mansour et al., 2008, 2009], where a target function mapping from X to Y is assumed.”
Ensemble Adversarial Training: Attacks and Defenses (2017, background): “…h ∈ H we deﬁne the empirical risk and the risk over the target distribution (or future adversary) We further deﬁne the average discrepancy distance (Mansour et al., 2009) between distributions A i and A ∗ with respect to a hypothesis space H as This quantity characterizes how “different” the…”
Adaptive Personalized Federated Learning (2020, methodology): “This approach is predominantly used in meta-learning methods such as MAML by Finn et al. (2017) or domain adaptation and transfer learning (Ben-David et al., 2010; Mansour et al., 2009; Pan and Yang, 2009).”
Learning Robust Representations by Projecting Superficial Statistics Out (2018, background): “Other papers bound worst-case performance under bounded shifts as measured by divergence measures on the train v.s. test distributions (Ben-David et al., 2010a; Mansour et al., 2009; Hu et al., 2016).”
Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions (2017, background): “…for finite samples, in addition to the transfer from source domains to target domains, we have to deal with the generalization from empirical to population distributions and from the covariate shift if P(A |C1 = 1) 6= P(A |C1 = 0) (see e.g. Mansour et al., 2009). obtain small generalization errors.”
Task-Free Continual Learning via Online Discrepancy Distance Learning (2022, positive): “Compared to the GB used in the domain adaptation [29], Theorem 1 provides an explicit way to measure the gap between the model’s prediction and the true labels in each training step (Ti).”
Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment (2019, background): “Existing theories without assumptions of contained support (Ben-David et al., 2007, 2010a; Ben-David & Urner, 2014; Mansour et al., 2009; Cortes & Mohri, 2011) do not exhibit this property since (i) when applied to the input space, their results are not concerned with domain-adversarial learning as…”
Simultaneous Deep Transfer Across Domains and Tasks (2015, background): “In fact, the size of a domain shift is often measured by the distance between the source and target subspace representations [5, 11, 21, 25, 27].”