Multiple Source Adaptation and the Rényi Divergence
By Yishay Mansour, Mehryar Mohri, Afshin Rostamizadeh. 2009
156 citations recorded by the source corpus.
This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in several directions. We extend previous multiple source loss guarantees based on distribution weighted combinations to arbitrary target distributions P, not necessarily mixtures of the source distributions, analyze both known and unknown target distribution cases, and prove a lower bound. We further extend our bounds to deal with the case where the learner receives an approximate distribution for each source instead of the exact one, and show that similar loss guarantees can be achieved depending on the divergence between the approximate and true distributions. We also analyze the case where the labeling functions of the source domains are somewhat different. Finally, we report the results of experiments with both an artificial data set and a sentiment analysis task, showing the performance benefits of the distribution weighted combinations and the quality of our bounds based on the Renyi divergence.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Information Theory, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mingsheng Long
- Michael I. Jordan
- Qiang Yang
- Sergey Levine
- Percy Liang
- Dacheng Tao
- Trevor Darrell
In Representation & Self-Supervised Learning
- Mingsheng Long
- Mehryar Mohri
- Jianmin Wang
- Qiang Yang
- Michael I. Jordan
- Kate Saenko
- Judy Hoffman
- Corinna Cortes
In Information Theory
- Mehryar Mohri
- Ningshan Zhang
- Jayadev Acharya
- Himanshu Tyagi
- Judy Hoffman
- Ananda Theertha Suresh
- Jieping Ye
- Lei Zhang
In Signal Processing
- Mehryar Mohri
- Fei Sha
- Boqing Gong
- Bernhard Schölkopf
- Ningshan Zhang
- Judy Hoffman
- Mingsheng Long
- Shiliang Sun
Citations in the corpus
- Multiple-Source Adaptation with Domain Classifiers (2020, methodology): “…information available to the learner, the number of source domains, and the amount of labeled and unlabeled data available from the target domain [Mansour et al., 2008, 2009a, Hoffman et al., 2018, Pan and Yang, 2010, Muandet et al., 2013, Xu et al., 2014, Hoffman et al., 2012, Gong et al.,…”
- A Theory of Multiple-Source Adaptation with Limited Target Labeled Data (2020, background): “It was shown by Mansour et al. (2009c) (see also Cortes and Mohri (2011)) that the discrepancy is the appropriate divergence between distributions in adaptation.”
- Agnostic Federated Learning (2019, background): “…the problem of transfer learning or domain adaptation from a single source to a known target domain (Ben-David, Blitzer, Crammer, and Pereira, 2006; Mansour, Mohri, and Rostamizadeh, 2009b; Cortes and Mohri, 2014; Cortes, Mohri, and Mu˜noz Medina, 2015), either through unsupervised adaptation…”
- Algorithms and Theory for Multiple-Source Adaptation (2018, methodology): “This is a more general setup than the deterministic scenario in [Mansour et al., 2008, 2009], where a target function mapping from X to Y is assumed.”
- Rényi Divergence and Kullback-Leibler Divergence (2012, background): “n a stationary Markov chain to the stationary distribution [1], and still other applications of R´enyi divergence can be found, for instance, in hypothesis testing [11], in multiple source adaptation [12] and in ranking of images [13]. Tim van Erven (tim@timvanerven.nl) is with the Departement de´ M”
- Theoretical Analysis of Domain Adaptation with Optimal Transport (2016, background): “[16] derives the domain adaptation bounds in multisource scenario by assuming that the good hypothesis can be learned as a weighted convex combination of hypothesis from all the sources available.”
- FedBoost: Communication-Efficient Algorithms for Federated Learning (2020, unknown): “As stated in Section 1.1, unlike MSDA, we show that such ensembles are optimal for density estimation.”
- A theory of learning from different domains (2010, background): “More recently, Mansour et al. (2009a, 2009b) introduced a theoretical model for the “multiple source adaptation problem.””
- On the Value of Target Data in Transfer Learning (2020, background): “A related line of work considers information-theoretic measures such as KL-divergence or Renyi divergence [11, 12] but has received relatively less attention.”
- Local Rademacher Complexity-based Learning Guarantees for Multi-Task Learning (2016, background): “Also, in the context of domain adaptation, similar considerations are examined by [41, 43, 42, 21, 66, 44] and [22].”
- Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices (2019, unknown): “Rényi divergence has been useful in many applications, including for the exponential mechanism in differential privacy [27, 1, 12, 52], lattice-based cryptography [4], information-theoretic encryption [35], variational inference [41], machine learning [32, 50], information theory and statistics [20,”
- Machine Learning Paradigms for Speech Recognition: An Overview (2013, unknown): “[199], [200]”
- Connecting the Dots with Landmarks: Discriminatively Learning Domain-Invariant Features for Unsupervised Domain Adaptation (2013, methodology): “With the reduced discrepancy between P S (X) and Q T (X), we can apply the analysis in (Mansour et al., 2009b) (Lemma 1) to show that classifiers applied to Q T (X) attain a smaller generalization error bound than those applied to P T (X).”
- Bridging Theory and Algorithm for Domain Adaptation (2019, background): “Domain Adaptation Theory The earliest theoretical work in this field was conducted by Ben-David et al. (2007).”
- On Learning Invariant Representations for Domain Adaptation (2019, background): “…particularly favorable in the analysis of domain adaptation with binary classification problems, and it had also been generalized to the discrepancy distance (Cortes et al., 2008; Mansour et al., 2009a;b; Cortes & Mohri, 2014) for general loss functions, including the one for regression problems.”