Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling

By Xiaochuang Han, Jacob Eisenstein. 2019

185 citations recorded by the source corpus.

Contextualized word embeddings such as ELMo and BERT provide a foundation for strong performance across a wide range of natural language processing tasks by pretraining on large corpora of unlabeled text. However, the applicability of this approach is unknown when the target domain varies substantially from the pretraining corpus. We are specifically interested in the scenario in which labeled data is available in only a canonical source domain such as newstext, and the target domain is distinct from both the labeled and pretraining texts. To address this scenario, we propose domain-adaptive fine-tuning, in which the contextualized embeddings are adapted by masked language modeling on text from the target domain. We test this approach on sequence labeling in two challenging domains: Early Modern English and Twitter. Both domains differ substantially from existing pretraining corpora, and domain-adaptive fine-tuning yields substantial improvements over strong BERT baselines, with particularly impressive results on out-of-vocabulary words. We conclude that domain-adaptive fine-tuning offers a simple and effective approach for the unsupervised adaptation of sequence labeling to difficult new domains.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, methodology): “In the presence of domain shift between train and test data distributions of the same task, domain-adaptive pretraining (DAPT) is sometimes used to describe what we term TAPT (Logeswaran et al., 2019; Han and Eisenstein, 2019).”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “Weak performance in older texts has been noted in NLP for historical documents (Yang and Eisenstein, 2016; Han and Eisenstein, 2019).”
PADA: A Prompt-based Autoregressive Approach for Adaptation to Unseen Domains (2021, methodology): “Later on, Han and Eisenstein (2019) presented a pre-training method, followed by Ben-David et al. (2020) who introduced PERL , a pivot-based variant for pre-training contextual word embeddings.”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, background): “, 2020) SSL 3 (Rotman and Reichart, 2019) Deep self-training 3 (Han and Eisenstein, 2019) AdaptaBERT 3 3 (Li et al.”
Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives (2022, background): “AdaptBERT (Han and Eisenstein, 2019) performs domain-adaptive fine-tuning to adapt contextualized embedding by masked language modeling from the target domain.”
PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains (2021, methodology): “Borrowing terminology from Liu et al. (2021a), our approach falls under the “Prompt+LM Tuning” training strategy (Liu et al., 2021b; Han et al., 2021).”
UDALM: Unsupervised Domain Adaptation through Language Modeling (2021, background): “This is consistent with previous works on supervised (Gururangan et al., 2020; Xu et al., 2019; Sun et al., 2019) and unsupervised settings (Han and Eisenstein, 2019; Du et al., 2020).”
Bridging the domain gap in cross-lingual document classification (2019, methodology): “It is natural to use unsupervised domain data for this task, when available (Lee et al. (2019); Han & Eisenstein (2019)).”