Domain Adaptation for Statistical Classifiers

By Hal Daumé Iii, Daniel Marcu. 2006

920 citations recorded by the source corpus.

The most basic assumption used in statistical learning theory is that training data and test data are drawn from the same underlying distribution. Unfortunately, in many applications, the "in-domain" test data is drawn from a distribution that is related, but not identical, to the "out-of-domain" distribution of the training data. We consider the common case in which labeled out-of-domain data is plentiful, but labeled in-domain data is scarce. We introduce a statistical formulation of this problem in terms of a simple mixture model and present an instantiation of this framework to maximum entropy classifiers and their linear chain counterparts. We present efficient inference algorithms for this special case based on the technique of conditional expectation maximization. Our experimental results show that our approach leads to improved performance on three real world tasks on four different data sets from the natural language processing domain.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, neutral): “Domain adaptation aims to address the mismatch in data distribution between training and test domains (Daume III & Marcu, 2006; Quionero-Candela et al., 2009; Koh et al., 2021).”
On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products (2016, background): “Further, it may be that the distribution the samples actually come from cannot be known, precluding the use of covariate shift [14] and domain adaptation techniques [15].”
Few-Shot Adversarial Domain Adaptation (2017, methodology): “Finally, some use regularization to improve the fit on the target domain [4, 1, 62, 10, 3, 8].”
Learning Representations for Counterfactual Inference (2016, background): “In the contextual bandit setting, there is typically an algo-rithm which determines the choice of the action t given the context x .”
Frustratingly Easy Domain Adaptation (2007, methodology): “Daumé III and Marcu (2006) provide empirical evidence on four datasets that the PRIOR model outperforms the baseline approaches.”
Multiple Source Adaptation and the Rényi Divergence (2009, background): “This problem arises in a variety of natural language processing tasks such parsing, statistical language modeling, text classification [15, 7, 16, 9, 10, 4, 6], or speech processing [11, 8, 14] and computer vision [13] tasks, as well as in many other applications.”
Unified Deep Supervised Domain Adaptation and Generalization (2017, background): “The third one regularizes a classifier trained on a source distribution to work well on a target distribution [4, 1, 66, 15, 3, 12].”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “named entity recognition and opinion extraction across different text corpora [59, 161, 15, 276], multilanguage text classification [167, 275, 274], sentiment analysis [35, 100], WiFi-based localization [159], speech recognition across different speakers [132, 175], object recognition in images acqu”
Demographic Factors Improve Classification Performance (2015, background): “1Apart from the demographic factors, other factors such as mood, interpersonal relationship, authority, language attitude, etc. contribute to our perception of language.”
Generalizing to Unseen Domains via Adversarial Data Augmentation (2018, neutral): “Standard models that perform well when evaluated on the validation dataset—usually collected from the same population as the training dataset—often perform poorly on populations different from that of the training data [15, 3, 1, 32, 38].”
Tagging Performance Correlates with Author Age (2015, background): “Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers), pages 483–488, Beijing, China, July 26-31, 2015. c©2015 Association for Computational Linguistics”
Glen, Glenda or Glendale: Unsupervised and Semi-supervised Learning of English Noun Gender (2009, methodology): “This is like the competitive Feats domain-adaptation system in Daumé III and Marcu (2006).”
Unsupervised Domain Adaptation by Domain Invariant Projection (2013, background): “In the former category, modifications of Support Vector Machines (SVM) [12, 3] and other statistical classifiers [10] have been proposed to exploit the availability of labeled and unlabeled data from the target domain.”
A Literature Survey on Domain Adaptation of Statistical Classifiers (2007, background): “The domain adaptation problem, especially domain adaptation in natural language processing, started gaining much attention very recently [Daumé III and Marcu, 2006, Blitzer et al., 2006, Ben-David et al., 2007, Daumé III, 2007, Satpal and Sarawagi, 2007].”
SAR Image Classification Using Few-Shot Cross-Domain Transfer Learning (2019, background): “Other common discrepancy measures such as KL divergence and Jensen-Shannon divergence can be used for a broader range of domain adaptation problems [7].”