Continual Training of Language Models for Few-Shot Learning

By Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, Bin Liu. 2022

45 citations recorded by the source corpus.

Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. Adapting or posttraining an LM using an unlabeled domain corpus can produce even better performance for end-tasks in the domain. This paper proposes the problem of continually extending an LM by incrementally post-train the LM with a sequence of unlabeled domain corpora to expand its knowledge without forgetting its previous skills. The goal is to improve the few-shot end-task learning in these domains. The resulting system is called CPT (Continual PostTraining), which to our knowledge, is the first continual post-training system. Experimental results verify its effectiveness.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Bin Liu
4. Tatsuya Konishi
5. Liyuan Wang
6. Gyuhak Kim
7. Xingxing Zhang
8. Zixuan Ke

In Computer Vision and Pattern Recognition

1. Hang Su
2. Jun Zhu
3. Bin Liu
4. Liyuan Wang
5. Xingxing Zhang
6. Gyuhak Kim
7. Zixuan Ke
8. Tatsuya Konishi

In Large Language Models

1. Bin Liu
2. Liyuan Wang
3. Tatsuya Konishi
4. Jun Zhu
5. Gyuhak Kim
6. Hang Su
7. Haowei Lin
8. Xingxing Zhang

In Meta & Continual Learning

1. Liyuan Wang
2. Bin Liu
3. Xingxing Zhang
4. Gyuhak Kim
5. Hang Su
6. Jun Zhu
7. Tatsuya Konishi
8. Haowei Lin

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The third is continual pre-training (CPT) or continual meta-training.”
• Continual Learning of Natural Language Processing Tasks: A Survey (2022, methodology): “…summarization and information extraction), which is usually supervised [Ke et al., 2021a; Monaikul et al., 2021; Qin and Joty, 2022], or a corpus domain used to further pre-train or post-train an LM, which is usually unsupervised [Gururangan et al., 2021; Ke et al., 2022; Scialom et al., 2022].”
• Continual Pre-training of Language Models (2023, background): “However, CPT uses the parameter-isolation approach to learn and protect each task, which is weak (see Sec.”
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, unknown): “Various training strategies have been proposed to address the problem of CF in continual learning (Riemer et al., 2019; Buzzega et al., 2020; Ke et al., 2022; Chen et al., 2022; Luo et al., 2023a).”