A theory of learning from different domains

By Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando C Pereira, Jennifer Wortman Vaughan. 2010

4,172 citations recorded by the source corpus.

Discriminative learning methods for classification perform well when training and test data are drawn from the same distribution. Often, however, we have plentiful labeled training data from a source domain but wish to learn a classifier which performs well on a target domain with a different distribution and little or no labeled training data. In this work we investigate two questions. First, under what conditions can a classifier trained from source data be expected to perform well on target data? Second, given a small amount of labeled target data, how should we combine it during training with the large amount of labeled source data to achieve the lowest target error at test time?We address the first question by bounding a classifier’s target error in terms of its source error and the divergence between the two domains. We give a classifier-induced divergence measure that can be estimated from finite, unlabeled samples from the domains. Under the assumption that there exists some hypothesis that performs well in both domains, we show that this quantity together with the empirical source error characterize the target error of a source-trained classifier.We answer the second question by bounding the target error of a model which minimizes a convex combination of the empirical source and target errors. Previous theoretical work has considered minimizing just the source error, just the target error, or weighting instances from the two domains equally. We show how to choose the optimal combination of source and target error as a function of the divergence, the sample sizes of both domains, and the complexity of the hypothesis class. The resulting bound generalizes the previously studied cases and is always at least as tight as a bound which considers minimizing only the target error or an equal weighting of source and target errors.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Unbounded cache model for online language modeling with open vocabulary (2017, background): “Notable variants of transfer learning are multitask learning [10], domain adaptation [6], and curriculum learning [8].”
Learning Robust Global Representations by Penalizing Local Predictive Power (2019, background): “While some methods have justiﬁed domain-adversarial learning by appealing to theoretical bounds due to Ben-David et al. (2010a), the theory does not in fact guarantee generalization (recently shown by Johansson et al. (2019) and Wu et al. (2019)) and sometimes guarantees failure.”
Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning (2021, background): “Domain adaptation methods (DA) (Ben-David et al., 2010) attempt to alleviate the distribution mismatch between source and target domains.”
Unsupervised Domain Adaptation by Backpropagation (2014, methodology): “Relation toH∆H-distance In this section we give a brief analysis of our method in terms ofH∆H-distance (Ben-David et al., 2010; Cortes & Mohri, 2011) which is widely used in the theory of nonconservative domain adaptation.”
Meta-learning with differentiable closed-form solvers (2018, background): “Over the years the goal of adapting learners to different tasks has also been studied under the umbrellas of multi-task learning [30, 31] and domain adaptation [32].”
Domain-Adversarial Training of Neural Networks (2015, methodology): “The theoretical foundation of the DANN algorithm is the domain adaptation theory of Ben-David et al. (2006, 2010).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “, 2018), and covariate and label shifts (Ben-David et al., 2010; Shimodaira, 2000; Lipton et al., 2018).”
Understanding Black-box Predictions via Influence Functions (2017, background): “Domain mismatch — where the training distribution does not match the test distribution — can cause models with high training accuracy to do poorly on test data (Ben-David et al., 2010).”
Learning Transferable Features with Deep Adaptation Networks (2015, methodology): “To approach this problem, many existing methods aim to bound the target error by the source error plus a discrepancy metric between the source and the target (Ben-David et al., 2010).”
Advances and Open Problems in Federated Learning (2019, background): “Given a global model that performs reasonably well, what is the best way to personalize it? In nonfederated learning, researchers often use fine-tuning, transfer learning, domain adaptation [284, 115, 56], or interpolation with a personal local model.”
Marginalized Denoising Autoencoders for Domain Adaptation (2012, background): “Domain adaptation (Ben-David et al., 2009; Huang et al., 2007; Weinberger et al., 2009; Xue et al., 2008) aims to generalize a classiﬁer that is trained on a source domain, for which typically plenty of training data is available, to a target domain, for which data is scarce.”
A Theory of Multiple-Source Adaptation with Limited Target Labeled Data (2020, background): “, 2008) and (Ben-David et al., 2010), where the authors presented an analysis in terms of a dA-distance, including VC-dimension learning bounds for the zero-one loss.”
Adaptive Personalized Federated Learning (2020, methodology): “Moreover, utilizing the multi-domain learning theory (Ben-David et al., 2010), we provide a theoretical analysis of the generalization bound for proposed personalization.”
Certifying Some Distributional Robustness with Principled Adversarial Training (2017, background): “In many systems, robustness to changes in the data-generating distribution P0 is desirable, whether they be from covariate shifts, changes in the underlying domain (Ben-David et al., 2010), or adversarial attacks (Goodfellow et al.”
The Variational Fair Autoencoder (2015, methodology): “Briefly, Proxy Adistance is an approximation to the H-divergence measure of domain distinguishability proposed in Kifer et al. (2004) and Ben-David et al. (2007; 2010).”
Generalization in anti-causal learning (2018, background): “Even though conditions for ﬁnite sample generalization bounds on different domains, e.g., (Ben-David et al., 2009), are rarely considered, let alone quantiﬁed in applications, the “anything can be learned” paradigm taught us to maintain high expectations with respect to learnability.”
Learning Robust Representations by Projecting Superficial Statistics Out (2018, background): “Other papers bound worst-case performance under bounded shifts as measured by divergence measures on the train v.s. test distributions (Ben-David et al., 2010a; Mansour et al., 2009; Hu et al., 2016).”
Domain Adaptation with Conditional Transferable Components (2016, methodology): “where d1(p(X, Y ), pT (X, Y )) is theL(1) or variation divergence defined in (Ben-David et al., 2010).”