Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

By Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew O. Arnold, Xiang Ren. 2021

153 citations recorded by the source corpus.

Pretrained language models (PTLMs) are typically learned over a large, static corpus and further fine-tuned for various downstream tasks. However, when deployed in the real world, a PTLM-based model must deal with data distributions that deviates from what the PTLM was initially trained on. In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologically-ordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithms, and keep track of the downstream task performance (after fine-tuning). We evaluate PTLM’s ability to adapt to new corpora while retaining learned knowledge in earlier corpora. Our experiments show distillation-based approaches to be most effective in retaining downstream performance in earlier domains. The algorithms also improve knowledge transfer, allowing models to achieve better downstream performance over latest data, and improve temporal generalization when distribution gaps exist between training and evaluation because of time. We believe our problem formulation, methods, and analysis will inspire future studies towards continual pretraining of language models.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “Recently, researchers have also conducted a series of empirical studies on the continual learning of PLMs (Wu et al., 2021; Jin et al., 2021).”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, methodology): “Most prior work in this space has focused on continual learning in LMs (Jin et al., 2021) or learning disparate tasks (de Masson d'Autume et al., 2019; Huang et al., 2021).”
TimeLMs: Diachronic Language Models from Twitter (2022, methodology): “In addition to these approaches to specialize language models, there have been similar temporal adaptation analyses to the one presented in our paper (Agarwal and Nenkova, 2021; Jin et al., 2021).”
Continual-T0: Progressively Instructing 50+ Tasks to Language Models Without Forgetting (2022, background): “Jin et al. (2021) show Continual Learning algorithms are effective for knowledge preservation.”
Can Foundation Models Wrangle Your Data? (2022, background): “Recent work on lifelong pretraining has identified ways of continually updating FMs in order to adapt them to emerging data [36, 77].”
Fine-tuned Language Models are Continual Learners (2022, background): “Jin et al. (2021) show CL algorithms are effective for knowledge preservation.”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “Extensive studies have not only demonstrated the necessity of CPT for improved downstream performance [35, 71, 95, 96, 100, 184], but also shown that when distributional shifts are gradual [95, 278] or somewhat correlated [71], CPT can effectively help model generalize to unseen data.”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, methodology): “Large Language Models (LMs) pretrained on a vast amount of text corpus have shown to be highly effective when ﬁnetuned or prompted to perform various downstream tasks (Raffel et al., 2019; Brown et al., 2020; Sanh et al., 2022; Wei et al., 2022).”
Continual Pre-training of Language Models (2023, methodology): “Jin et al. (2021) evaluated several existing CL techniques in a similar setting as DAS and performed analyses on dealing with CF.”