Algorithms and Theory for Multiple-Source Adaptation

By Judy Hoffman, Mehryar Mohri, Ningshan Zhang. 2018

191 citations recorded by the source corpus.

We present a number of novel contributions to the multiple-source adaptation problem. We derive new normalized solutions with strong theoretical guarantees for the cross-entropy loss and other similar losses. We also provide new guarantees that hold in the case where the conditional probabilities for the source domains are distinct. Moreover, we give new algorithms for determining the distribution-weighted combination solution for the cross-entropy loss and other losses. We report the results of a series of experiments with real-world datasets. We find that our algorithm outperforms competing approaches by producing a single robust model that performs well on any target mixture distribution. Altogether, our theory, algorithms, and empirical results provide a full solution for the multiple-source adaptation problem with very practical benefits.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Statistical Learning Theory

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “Can we guarantee that the model hFL learned via federated learning is at least as accurate as hk when used for client k? Can we quantify how much improvement can be expected via federated leaning? And can we develop personalization strategies with theoretical guarantees that at least match the perfo”
Multiple-Source Adaptation with Domain Classifiers (2020, methodology): “However, this discriminative solution admits several significant advantages and, with ideal conditional probabilities, this proposed algorithm also has the same optimal guarantees as the ideal density estimation solution of [Hoffman et al., 2018].”
A Theory of Multiple-Source Adaptation with Limited Target Labeled Data (2020, background): “In order to provide meaningful bounds and improve upon (1), following (Mansour et al., 2009c; Hoffman et al., 2018), we assume that the target distribution is close to some convex combination of sources in the discrepancy measure, that is, we assume that there is a λ ∈ ∆p such that discH(D0,Dλ) is s”
Agnostic Federated Learning (2019, methodology): “Proof The proof is an extension of the standard proofs for Rademacher complexity generalization bounds (Koltchinskii and Panchenko, 2002; Mohri et al., 2018).”
Moment Matching for Multi-Source Domain Adaptation (2018, background): “Previous theoretical bounds [1, 14, 49] on the target error are based on the H∆H-divergence between the source and target domains.”
Multiple-source adaptation theory and algorithms (2020, background): “This work is an extended version of the conference paper of [30], with more explicit proofs for some results and with entirely new sections.”
Adversarial target-invariant representation learning for domain generalization (2019, background): “Following Hoﬀman et al. (2018), we assume that all examples that appear at test time can be explained by a mixture of the source domains, i.e., any target domain D can be written as π i,j ∈ ∆ N S .”
Robust Learning from Untrusted Sources (2019, background): “Another related line of work is the one of (Mansour et al., 2009; Hoffman et al., 2018), who provide guarantees for a classifier learned on data from N domains on any target distribution that is a mixture of the distributions of the sources.”
A Survey of Unsupervised Deep Domain Adaptation (2018, methodology): “[227] survey multi-source domain adaptation, and since then a number of other methods [28, 88, 95, 157, 184, 192, 256, 259, 281] have been developed for this case.”
Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results (2020, methodology): “Unsupervised domain adaptation methods have been extensively studied [17], [18], [19], [20], [21], [22], [23], [24].”
Distributionally Robust Language Modeling (2019, background): “More recent work consider errors in estimating the target domain (Hoffman et al., 2018) and derive learning bounds with respect to such errors.”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, methodology): “It has been successfully applied to unsupervised domain adaptation [16, 53] and later extended to pixel-wise adaptation [6, 24, 10] and multi-source adaptation [25].”
Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization (2021, background): “These results have been generalized to include multiple sources (Blitzer et al., 2007; Hoffman et al., 2018; Mansour et al., 2008).”
Domain Aggregation Networks for Multi-Source Domain Adaptation (2019, background): “When there are a lot of source domains, MDMN will be noticeably slower than MDAN and DARN.”
FedBoost: Communication-Efficient Algorithms for Federated Learning (2020, unknown): “…related to that of multiple source domain adaptation (MSDA), first formalized and analyzed theoretically by Mansour, Mohri, and Rostamizadeh (2009b;a) and Hoffman et al. (2018) and later studied for various applications such as object recognition (Hoffman et al., 2012; Gong et al., 2013a;b).”
Test-Time Training for Out-of-Distribution Generalization (2019, background): “Previous works make the source and target distributions broader with multiple and evolving sources and targets without fundamentally address this problem (Hoffman et al., 2018; 2012; 2014).”
Test-Time Training with Self-Supervision for Generalization under Distribution Shifts (2019, background): “Prior work try to anticipate broader distributions by using multiple and evolving domains (Hoffman et al., 2018; 2012; 2014).”