Pretrained Language Model in Continual Learning: A Comparative Study

By Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, Gholamreza Haffari. 2022

77 citations recorded by the source corpus.

Subfields: Large Language Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Maosong Sun
2. Zhiyuan Liu
3. Xiang Ren
4. Liyuan Wang
5. Peng Li
6. Jun Zhu
7. Arthur Douillard
8. Xindong Wu

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Xingxing Zhang
5. Liyuan Wang
6. Hang Su
7. Qiwen Wang
8. Timothée Lesort

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “received increasing attentions, which differs from continual learning in vision domains for two aspects: (1) the greater efforts in pre-training, including both upstream [152], [187], [411] and downstream [114], [466] continual learning; and (2) the specialized architectures of transformer.”
• Continual Pre-Training Mitigates Forgetting in Language and Vision (2022, background): “Pre-trained models are widespread also in CL (Mehta et al., 2021; Ramasesh et al., 2021; Wu et al., 2021), where they are mostly used to conveniently initialize the model weights before learning from the non-stationary stream of data.”
• ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “Recently, researchers have also conducted a series of empirical studies on the continual learning of PLMs (Wu et al., 2021; Jin et al., 2021).”
• Deep Class-Incremental Learning: A Survey (2023, unknown): “, semantic segmentation [164], [165], [166], [167], [174], object detection [175], [176], [177], [178], [179], [180], [181], video understanding [95], [133], medical surgery [182], [183], language model [184], [185], [186], [187], language-vision model [188], [189], etc.”
• Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “Wu et al. (2022) present a comparative study of algorithms in the context of continual fine-tuning over NLP tasks.”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, methodology): “To better unify LLMs and KGs, many advanced techniques need to be incorporated, such as multi-modal learning [261], graph neural network [262], and continuous learning [263].”
• Class-Incremental Learning: A Survey (2023, background): “…object detection [137], [242], [243], [244], [245], [246], [247], video understanding [91], [161], medical surgery [248], [249], language model [250], [251], [252], [253], language-vision model [217], [254], etc. Specifically, object detection and semantic segmentation are two popular vision…”
• Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “Wu et al. (2022) investigated large NLP models (both encoders-decoders and decoders based) and concluded on the existence of robust early layers thanks to the SSL pre-training but remarked that middle and late layers still heavily suffered from forgetting.”
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, unknown): “As a result, they propose using an optimal linear classifier of learned tasks to measure performance, with changes considered as a surrogate to quantify CF. Similarly, Wu et al. (2023) [5] employ layer-wise and task-wise probing to analyze CF in each layer for previously learned tasks.”