Theoretical Analysis of Domain Adaptation with Optimal Transport

By Ievgen Redko, Amaury Habrard, Marc Sebban. 2016

222 citations recorded by the source corpus.

Domain adaptation (DA) is an important and emerging field of machine learning that tackles the problem occurring when the distributions of training (source domain) and test (target domain) data are similar but different. This kind of learning paradigm is of vital importance for future advances as it allows a learner to generalize the knowledge across different tasks. Current theoretical results show that the efficiency of DA algorithms depends on their capacity of minimizing the divergence between source and target probability distributions. In this paper, we provide a theoretical study on the advantages that concepts borrowed from optimal transportation theory [17] can bring to DA. In particular, we show that the Wasserstein metric can be used as a divergence measure between distributions to obtain generalization guarantees for three different learning settings: (i) classic DA with unsupervised target data (ii) DA combining source and target labeled data, (iii) multiple source DA. Based on the obtained results, we motivate the use of the regularized optimal transport and provide some algorithmic insights for multi-source domain adaptation. We also show when this theoretical analysis can lead to tighter inequalities than those of other existing frameworks. We believe that these results open the door to novel ideas and directions for DA.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Mathematical Optimization

Citations in the corpus

Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, background): “rnt distributions are the exact approximations to their target distributions while learning every given task. In the following we extend the theoretical analysis on the domain adaptation problem from [53] (Theorem 2) in order to analyze how the knowledge learned by GRMs is lost during the lifelong l”
Generative Continual Concept Learning (2019, methodology): “We follow a standard PAC-learning style framework to analyze our algorithm [20] and using result from domain adaptation [26] to demonstrate the effectiveness of our algorithm.”
Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replay (2019, methodology): “We use existing theoretical results about using optimal transport within domain adaptation [Redko et al., 2017], to justify why our algorithm can tackle catastrophic forgetting.”
A Survey of Unsupervised Deep Domain Adaptation (2018, background): “A problem known as “optimal transport” was originally proposed for studying resource allocation such as finding an optimal way to move material from mines to factories [169, 192], but it can also be used to measure the distances between distributions.”
Learning Transferable Knowledge Through Embedding Spaces (2019, methodology): “, data points from both domains that belong to a single class geometrically form a cluster in the embedding, then a classifier learned using solely the labeled data from the source domain, generalizes well on data points from the target domain [104].”
SAR Image Classification Using Few-Shot Cross-Domain Transfer Learning (2019, methodology): “If we can train two deep encoders to map the data points from both domains into a shared embedding space such that both domains would have similar class-conditioned probability distributions in the embedding space, then a classifier trained using the source-domain data points in the shared embedding”
DeepJDOT: Deep Joint distribution optimal transport for unsupervised domain adaptation (2018, unknown): “Optimal transport [22–24] has been used in domain adaptation to learn the transformation between domains [4, 25, 26], with associated theoretical guarantees [27].”
Sequential Unsupervised Domain Adaptation through Prototypical Distributions (2020, methodology): “Redko et al. [42] prove Theorem 2 for a binary classiﬁcation setting in a joint training UDA setting.”
Deep Transfer Learning for Few-Shot SAR Image Classification (2019, methodology): “If both domains have similar 108 class-conditional probability distributions in the embedding space, then if we train a classifier network using 109 only the source-domain labeled data points from the shared embedding to the label space, it will also generalize 110 well on the target domain test dat”
On Localized Discrepancy for Domain Adaptation (2020, methodology): “Courty et al. (2017); Redko et al. (2017); Courty et al. (2017) consider using Wasserstein distance as surrogate of the parameterized discrepancy.”
A Review of Domain Adaptation without Target Labels (2019, background): “probability mass assigned to the i-th sample [136].”
Obtaining Fairness using Optimal Transport Theory (2018, background): “Similar inequalities in the framework of domain adaptation are given in (Redko et al., 2017).”
Wasserstein Distance Guided Representation Learning for Domain Adaptation (2017, background): “Genralization Bound (Redko, Habrard, and Sebban 2016) proved that the target error can be bounded by the Wasserstein distance for empirical measures.”
Learning a Domain-Invariant Embedding for Unsupervised Domain Adaptation Using Class-Conditioned Distribution Alignment (2019, methodology): “[28] provided theoretical guarantees for using a Wasserstein metric to address domain adaptation.”