Generalization Bounds for Domain Adaptation

By Lei Zhang, Jieping Ye. 2012

102 citations recorded by the source corpus.

In this paper, we provide a new framework to study the generalization bound of the learning process for domain adaptation. We consider two kinds of representative domain adaptation settings: one is domain adaptation with multiple sources and the other is domain adaptation combining source and target data. In particular, we use the integral probability metric to measure the difference between two domains. Then, we develop the specific Hoeffding-type deviation inequality and symmetrization inequality for either kind of domain adaptation to achieve the corresponding generalization bound based on the uniform entropy number. By using the resultant generalization bound, we analyze the asymptotic convergence and the rate of convergence of the learning process for domain adaptation. Meanwhile, we discuss the factors that affect the asymptotic behavior of the learning process. The numerical experiments support our results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Ensemble Adversarial Training: Attacks and Defenses (2017, background): “x attacks from some set of models. Domain Adaptation with multiple sources. We can draw a connection between Ensemble Adversarial Training and multiple-source Domain Adaptation (Mansour et al., 2009; Zhang et al., 2012). In Domain Adaptation, a model trained on data sampled from one or more source d”
Theoretical Analysis of Domain Adaptation with Optimal Transport (2016, background): “[27] provides a comparative analysis of discrepancy and integral metric based bounds and shows that the former are less tight.”
A New PAC-Bayesian Perspective on Domain Adaptation (2015, unknown): “The divergence-based approach has especially been explored to derive generalization bounds for domain adaptation ( e.g., Ben-David et al., 2006; 2010; Mansour et al., 2009; Li & Bilmes, 2007; Zhang et al., 2012).”
Generalized Transfer Subspace Learning Through Low-Rank Constraint (2014, background): “…and specifically, domain adaptation (Daumé III and Marcu 2006; Blitzer et al. 2006, 2011; Daumé 2007; Sun et al. 2011; Glorot et al. 2011; Chen et al. 2011; Lopez-Paz et al. 2012; Zhang et al. 2012) transforms the source domain by either manipulating its distribution or feature representation.”
Local Rademacher Complexity-based Learning Guarantees for Multi-Task Learning (2016, background): “Also, in the context of domain adaptation, similar considerations are examined by [41, 43, 42, 21, 66, 44] and [22].”
Robust Learning from Untrusted Sources (2019, result): “We note that a similar result appears as Theorem 5.2 in the arXiv version (Zhang et al., 2013) of the NIPS paper (Zhang et al., 2012).”
Multi-Domain Adversarial Learning (2019, background): “Furthermore, the target risk is upper-bounded by the empirical source risk, the empirical H -divergence between source(s) and target marginals, and the oracle DA risk (Ben-David et al., 2006; 2010; Zhang et al., 2012).”
Prompt-aligned Gradient for Prompt Tuning (2022, methodology): “We bound the generalization error of ProGrad by virtue of Rademacher Complexity [1] and the Theorem 6.2 in [53].”
Bridging Theory and Algorithm for Domain Adaptation (2019, background): “Mohri & Medina (2012); Zhang et al. (2012) proposed Y -disc for domain adaptation with partially labeled target data.”
Domain Generalization and Adaptation Using Low Rank Exemplar SVMs (2018, background): “It has been shown in the previous works [34], [35] that the error of fi on the test data can be bound as errt(fi) ≤ errs(fi) + dist(D i ,D) + ∆, where errt(fi) is the test error, errs(fi) is the training error of fi, dist(D i ,D) is the distance between two distributions under certain measurement, a”
Adaptation Based on Generalized Discrepancy (2019, methodology): “The discrepancy is a pivotal concept used in the analysis of several adaptation scenarios: the Y-discrepancy or integral probability metric (Zhang et al., 2012) was successfully used by Mohri and Muñoz (2012) to provide tight learning guarantees for the related task of learning with drifting…”
An introduction to domain adaptation and transfer learning (2018, background): “Nonetheless, it is possible to derive generalization error bounds if more is known on the relationship between S and T [4,29,43,144,16,215,78].”
Security and Privacy Issues in Deep Learning (2018, background): “Tramèr et al. [2017] introduces a connection between Ensemble adversarial training and multiple-source domain adaptation [Mansour et al. 2009; Zhang et al. 2012].”