Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

By Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith. 2020

3,085 citations recorded by the source corpus.

Language models pretrained on text from a wide variety of sources form the foundation of today’s NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task’s unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Overall, we consistently find that multi-phase adaptive pretraining offers large gains in task performance.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Challenges in Detoxifying Language Models (2021, methodology): “Prior work has considered various approaches towards reducing LM toxicity, either by fine-tuning a pre-trained LM (Gehman et al., 2020; Gururangan et al., 2020),”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, neutral): “DAPT We consider domain-adaptive pretraining (Gururangan et al., 2020, DAPT), i.e., ﬁnetun-ing LMs on nontoxic data.”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “This raises the question: can language models ever fully “forget” toxic pretraining data through further adaptation (Kirkpatrick et al., 2017; Gururangan et al., 2020)? The non-trivial amounts of toxicity generated by DAPT suggest that perhaps language models may be “memorizing” the toxicity in pret”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “To prevent the model from “forgetting” how to generate other languages, we use a strategy inspired by domain/task-adaptive pre-training (Howard and Ruder, 2018; Gururangan et al., 2020): We simply”
Measuring Mathematical Problem Solving With the MATH Dataset (2021, background): “Since pretraining data can greatly inﬂuence performance (Hernandez et al., 2021; Gururangan et al., 2020) and since mathematics is a small fraction of online text, we introduce a large and diverse mathematics pretraining corpus.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “Along similar lines, Keskar et al. (2019); Gururangan et al. (2020); Chan et al. (2021) adopt a setup of training class-conditional LMs where each LM accepts "control codes" as an additional input.”
Improving and Simplifying Pattern Exploiting Training (2021, methodology): “Specifically, we experiment with Task Adaptive Pre-training (TAPT) (Gururangan et al., 2020) and pre-train our base LM for 2500 batches on the unlabeled data of FewGLUE.”
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets (2021, background): “[18] show that pretraining a model to specific domains and tasks results in improved performance.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “Efforts on the data side have focused on intermediate ﬁne-tuning either by using unlabeled target domain data (Karouzos et al., 2021; Gururangan et al., 2020) or via labeled data from other tasks (Phang et al., 2018; Agha-janyan et al., 2021; Vu et al., 2020).”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, background): “For fair evaluation of zero-shot generalization, we investigate and collect diverse public Chinese NLP datasets with different task types.”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, background): “While in some cases, such as those presented in their paper, it might be plausible to generate counterfactual examples, in most cases in NLP it is still too hard to generate realistic texts with conditional generative models (Lin et al. 2017; Che et al. 2017; Rajeswar et al. 2017; Guo et al. 2018).”
Continual Training of Language Models for Few-Shot Learning (2022, methodology): “Post-training (a.k.a., domain-adaptive pre-training or pre-ﬁnetuning ) an LM with a large unlabeled domain corpus before end-task ﬁne-tuning in the domain achieves better results (Xu et al., 2019; Gururangan et al., 2020a) than directly ﬁne-tuning the LM.”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, methodology): “Some target at domain adaptation through continual pre-training (Gururangan et al., 2020), parameter-efﬁcient adapters (He et al., 2021) and sparse expert models (Gururangan et al., 2021).”