Information-Theoretical Learning of Discriminative Clusters for Unsupervised Domain Adaptation

By Yuan Shi, Fei Sha. 2012

252 citations recorded by the source corpus.

We study the problem of unsupervised domain adaptation, which aims to adapt classifiers trained on a labeled source domain to an unlabeled target domain. Many existing approaches first learn domain-invariant features and then construct classifiers with them. We propose a novel approach that jointly learn the both. Specifically, while the method identifies a feature space where data in the source and the target domains are similarly distributed, it also learns the feature space discriminatively, optimizing an information-theoretic metric as an proxy to the expected misclassification error on the target domain. We show how this optimization can be effectively carried out with simple gradient-based methods and how hyperparameters can be cross-validated without demanding any labeled data from the target domain. Empirical studies on benchmark tasks of object recognition and sentiment analysis validated our modeling assumptions and demonstrated significant improvement of our method over competing ones in classification accuracies.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Visual Domain Adaptation: A survey of recent advances (2015, methodology): “P x P x t s ^ h For instance, nonparametric density estimation [1], [13] and kernel mean match-based methods [14] have been proposed in the literature to directly estimate the ratio.”
Source Data-Absent Unsupervised Domain Adaptation Through Hypothesis Transfer and Labeling Transfer (2020, methodology): “feature representations and outputs of the classifier, since information maximization [26], [27] can encourage the classifier to assign disparate one-hot outputs to different target feature representations.”
Small Sample Learning in Big Data Era (2018, methodology): “Considering that model adaptation and data transportation (Section 4.2.5) are not independent, optimizing both the transformation and classifier parameters jointly was also developed in (Shi and Sha, 2012; Hoffman et al., 2013; Saito et al., 2018).”
Transfer Learning via Learning to Transfer (2018, methodology): “Even with experiences generated by single base al-gorithm, e.g., ITL or DIP, the L2T can still learn a reﬂection function that signiﬁcantly better ( p -value 0.05) decides what to transfer than using ITL or DIP directly.”
Generalized Source-free Domain Adaptation (2021, background): “The second term avoids the degenerated solution [34, 8], where the prediction of classes in the target data is highly imbalanced, by encouraging prediction balance.”
Cluster Alignment With a Teacher for Unsupervised Domain Adaptation (2019, methodology): “However, one limitation of [40] is that it adopts a simple linear transformation to learn the feature space, which cannot effectively extract high-order features from raw data (e.”
Domain Adaptation Techniques for EEG-Based Emotion Recognition: A Comparative Study on Two Public Datasets (2019, background): “SA, ITL, and KPCA are less sensitive to varying number of source dataset size.”
Semi-Generative Modelling: Covariate-Shift Adaptation with Cause and Effect Features (2018, methodology): “Various criteria have been used to measure such similarity, e.g., MMD (Pan et al., 2011), HSIC (Yan et al., 2017), mutual information with D (Shi and Sha, 2012), or performance of a domain classiﬁer (Ganin et al., 2016).”
Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation (2021, methodology): “To avoid the degenerated solution [8, 39] where the model predicts all data as some specific classes (and does not predict other classes for any of the target data), we encourage the prediction to be balanced.”
Subspace Interpolation via Dictionary Learning for Unsupervised Domain Adaptation (2013, background): “Shi and Sha [29] define an information-theoretic measure which balances between maximizing domain similarity and minimizing expected classification error on the target domain.”
A Review of Domain Adaptation without Target Labels (2019, methodology): “Other techniques of this type include information-theoretic learning, where the authors assume that classes across domains remain clustered in high-dimensional space [144].”
Unsupervised Domain Adaptation via Structurally Regularized Deep Clustering (2020, methodology): “To enforce the cluster assumption, conditional entropy minimization [20, 32] is widely used in the UDA community [11, 44, 45, 50, 51, 60, 64, 68].”
Transfer String Kernel for Cross-Context DNA-Protein Binding Prediction (2016, background): “For example, [23], [41] propose novel feature representations for the domain adaptation purpose.”
Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation (2020, methodology): “For this purpose, we adopt the information maximization (IM) loss (Krause et al., 2010; Shi & Sha, 2012; Hu et al., 2017), to make the target outputs individually certain and globally diverse.”
DASH-N: Joint Hierarchical Domain Adaptation and Feature Learning (2015, methodology): “An information-theoretic method for unsupervised domain adaptation was proposed by Shi and Sha [21] that attempted to find a common feature space, where the source and target data distributions are similar and the misclassification error is minimized.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “Another favorable clustering-based regularization is mutual information maximization, which tries to maximize the mutual information [286], [287] between inputs and the discrete labels as follows,”