A New PAC-Bayesian Perspective on Domain Adaptation

By Pascal Germain, Amaury Habrard, François Laviolette, Emilie Morvant. 2015

71 citations recorded by the source corpus.

We study the issue of PAC-Bayesian domain adaptation: We want to learn, from a source domain, a majority vote model dedicated to a target one. Our theoretical contribution brings a new perspective by deriving an upper-bound on the target risk where the distributions' divergence-- expressed as a ratio--controls the trade-off between a source error measure and the target voters' disagreement. Our bound suggests that one has to focus on regions where the source data is informative. From this result, we derive a PAC-Bayesian generalization bound, and specialize it to linear classifiers. Then, we infer a learning algorithm and perform experiments on real data.

Subfields: Artificial Intelligence, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical Learning Theory

Citations in the corpus

PAC-Bayesian Theory Meets Bayesian Inference (2016, neutral): “However, despite the amount of work dedicated to this statistical learning theory—many authors improved the initial results [9, 29, 33, 43, 48] and/or generalized them for various machine learning setups [4, 14, 20, 28, 38, 44, 45, 46]—it is mostly used as a frequentist method.”
On Localized Discrepancy for Domain Adaptation (2020, background): “In addition, Germain et al. (2016); Johansson et al. (2019) derive upper bounds for target error with divergences defined with the density ratio.”
Domain Generalization by Marginal Transfer Learning (2017, background): “Finally, there are several works that assume the existence of a predictor that achieves good performance on both source and target domains (Ben-David et al., 2007, 2010; Blitzer et al., 2008; Mansour et al., 2009a; Cortes et al., 2015; Germain et al., 2016).”
A Review of Domain Adaptation without Target Labels (2019, background): “bound was derived that is not dependent on the error of the ideal joint hypothesis, Instead, it utilizes a term, hT nS , describing the risk in the part of the target domain for which the source domain is uninformative [83]:”
Lifelong Learning with Weighted Majority Votes (2016, background): “They are also often considered in works that utilize PAC-Bayesian techniques [22, 12].”
User-friendly introduction to PAC-Bayes bounds (2021, methodology): “There is still a lot of work to do to address this practical problem, but an interesting approach is proposed in [80]: the authors use a technique called domain adaptation to allow the use of PAC-Bayes bounds in this context.”
An introduction to domain adaptation and transfer learning (2018, background): “Nonetheless, it is possible to derive generalization error bounds if more is known on the relationship between S and T [4,29,43,144,16,215,78].”