ELLE: Efficient Lifelong Pre-training for Emerging Data

By Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou. 2022

92 citations recorded by the source corpus.

Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of various sources may continuously grow. This requires PLMs to integrate the information from all the sources in a lifelong manner. Although this goal could be achieved by exhaustive pre-training on all the existing data, such a process is known to be computationally expensive. To this end, we propose ELLE, aiming at efficient lifelong pre-training for emerging data. Specifically, ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM’s width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentangle the versatile knowledge learned during pre-training and stimulate the proper knowledge for downstream tasks. We experiment ELLE with streaming data from 5 domains on BERT and GPT. The results show the superiority of ELLE over various lifelong learning baselines in both pre-training efficiency and downstream performances. The codes are publicly available at https://github.com/thunlp/ELLE.

Subfields: Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

1. Lei Shu
2. Tatsuya Konishi
3. Gyuhak Kim
4. Hu Xu
5. Bin Liu
6. Zixuan Ke

In Meta & Continual Learning

1. Zixuan Ke
2. Bin Liu
3. Gyuhak Kim
4. Hu Xu
5. Lei Shu
6. Tatsuya Konishi
7. Haowei Lin
8. Yijia Shao

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “[248], CID [265], PAGeR [415], LFPT5 [342], DnR [396], CLNMT [51], COKD [376]), experience replay (CFID [248], CID [265], ELLE [345], IDBR [172], MBPA++ [87], Meta-”
• Continual Training of Language Models for Few-Shot Learning (2022, background): “We tried to adapt ELLE for continual post-training by learning from a pre-trained RoBERTa but it fails to converge.”
• Knowledge Inheritance for Pre-trained Language Models (2021, unknown): “Another line of methods (Gong et al., 2019; Gu et al., 2021; Chen et al., 2022; Qin et al., 2022) proposes to pre-train larger PLMs progressively.”
• Continual Pre-training of Language Models (2023, background): “ELLE has to start from pre-training the LM itself rather than from a pre-trained LM like DAS.”