Domain Adaptation with Structural Correspondence Learning

By John Blitzer, Ryan T. McDonald, Fernando C Pereira. 2006

1,733 citations recorded by the source corpus.

Discriminative learning methods are widely used in natural language processing. These methods work best when their training and test data are drawn from the same distribution. For many NLP tasks, however, we are confronted with new domains in which labeled data is scarce or non-existent. In such cases, we seek to adapt existing models from a resource-rich source domain to a resource-poor target domain. We introduce structural correspondence learning to automatically induce correspondences among features from different domains. We test our technique on part of speech tagging and show performance gains for varying amounts of source and target training data, as well as improvements in target domain parsing accuracy using our improved tagger.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “Learning widely applicable representations of words has been an active area of research for decades, including non-neural (Brown et al., 1992; Ando and Zhang, 2005; Blitzer et al., 2006) and neural (Mikolov et al.”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, background): “, 2007), including sequence and part-of-speech tagging (Blitzer et al., 2006; Peng and Dredze, 2017).”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, background): “…language representations and features to the downstream tasks, which includes word-level representations (Brown et al., 1992; Ando & Zhang, 2005; Blitzer et al., 2006; Collobert & Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) and sentence-level representations (Kiros et al.,…”
Domain-Adversarial Training of Neural Networks (2015, background): “Over the years, a large part of the literature has focused mainly on linear hypothesis (see for instance Blitzer et al., 2006; Bruzzone and Marconcini, 2010; Germain et al., 2013; Baktashmotlagh et al., 2013; Cortes and Mohri, 2014).”
Marginalized Denoising Autoencoders for Domain Adaptation (2012, methodology): “We evaluate mSDA on the Amazon reviews benchmark data sets (Blitzer et al., 2006) together with several other algorithms for representation learning and domain adaptation.”
Learning Adversarially Fair and Transferable Representations (2018, background): “4 These papers discuss the bound on ∆DP (g) in terms of the H-divergence (Blitzer et al., 2006), which is simply the statistical distance ∆∗ up to a multiplicative constant.”
Few-Shot Adversarial Domain Adaptation (2017, neutral): “Finetuning, while simple to implement, has been found to be sub-optimal when compared to later techniques such as domain adaptation [5].”
Frustratingly Easy Domain Adaptation (2007, background): “A part-of-speech tagging problem on PubMed abstracts introduced by Blitzer et al. (2006). There are two domains: the source domain is the WSJ portion of the Penn Treebank and the target domain is PubMed.”
Efficient Learning of Domain-invariant Image Representations (2013, background): “Several authors considered learning feature representations for unsupervised and transfer learning [23], and for domain adaptation [18, 24].”
How Transferable are Neural Networks in NLP Applications? (2016, neutral): “, 2010), and structural correspondence learning (Blitzer et al., 2006; Prettenhofer and Stein, 2010).”
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments (2014, background): “Although several techniques exist that can learn to map hand-crafted features from one domain to another [4, 7, 18], it is in general non-trivial to come up with good features which generalize well across tasks, and even harder across different languages.”
A Comprehensive Survey on Transfer Learning (2019, methodology): “1) Feature selection: SCL first performs feature selection operations to obtain the pivot features.”
Learning to select data for transfer learning with Bayesian Optimization (2017, methodology): “Sentiment Analysis For sentiment analysis, we evaluate on the Amazon reviews dataset (Blitzer et al., 2006).”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, background): “First, we focus on language modelling—a foundational task that is used for many NLP systems today through LM pretraining—and propose a benchmark to measure progress in this direction.”
Unified Deep Supervised Domain Adaptation and Generalization (2017, background): “Since this practice is known for leading to suboptimal performance, techniques such as domain adaptation [6] and/or domain generalization [5] have been developed to address the issue.”
Bayesian Multitask Learning with Latent Hierarchies (2009, background): “Blitzer, McDonald and Pereira [4] present another approach to this “unsupervised” setting of domain adaptation that makes use of prior knowledge of features that are expected to behave similarly across domains.”
Learning with Augmented Features for Heterogeneous Domain Adaptation (2012, methodology): “Based on structural correspondence learning (Blitzer et al., 2006), two method-s (Prettenhofer & Stein, 2010; Wei & Pal, 2010) were recently proposed to extract the so-called pivot features from the source and target domains, which is speciﬁcally designed for the cross-language text classi-ﬁcation…”