Continual Learning of Natural Language Processing Tasks: A Survey

By Zixuan Ke, Bin Liu. 2022

116 citations recorded by the source corpus.

Continual learning (CL) is a learning paradigm that emulates the human capability of learning and accumulating knowledge continually without forgetting the previously learned knowledge and also transferring the learned knowledge to help learn new tasks better. This survey presents a comprehensive review and analysis of the recent progress of CL in NLP, which has significant differences from CL in computer vision and machine learning. It covers (1) all CL settings with a taxonomy of existing techniques; (2) catastrophic forgetting (CF) prevention, (3) knowledge transfer (KT), which is particularly important for NLP tasks; and (4) some theory and the hidden challenge of inter-task class separation (ICS). (1), (3) and (4) have not been included in the existing survey. Finally, a list of future directions is discussed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Natural Language Processing, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yunchao Wei
2. Guoliang Kang
3. Gengwei Zhang
4. Liyuan Wang
5. Ling Chen
6. Xuanjing Huang
7. Rui Zheng
8. Tao Gui

In Computer Vision and Pattern Recognition

1. Yunchao Wei
2. Guoliang Kang
3. Gengwei Zhang
4. Liyuan Wang
5. Ling Chen
6. Xuanjing Huang
7. Tao Gui
8. Yi Yang

In Radiology, Nuclear Medicine and Imaging

1. Liyuan Wang
2. Yunchao Wei

In Natural Language Processing

1. Liyuan Wang
2. Xuanjing Huang
3. Tao Gui
4. Yunchao Wei
5. Xipeng Qiu
6. Pengfei Liu
7. Michael Crawshaw
8. Yonatan Belinkov

In Meta & Continual Learning

1. Yunchao Wei
2. Liyuan Wang
3. Gengwei Zhang
4. Ling Chen
5. Guoliang Kang

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “In contrast, the latest surveystypicallycollateonlyalocalaspectofcontinuallearning, withrespecttoitsbiologicalunderpinnings[1],[3],[9],[10], specializedsettingsforvisualclassification[11],[12],[13],[14], andspecificextensionsforNLP[15],[16]orRL[17].”
• SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, background): “Inspired by the techniques of using pre-trained knowledge in NLP, L2P [45] employed an additional set of learnable parameters called “prompts” that dynamically instruct a pre-trained representation layer to learn incremental tasks.”
• Orthogonal Subspace Learning for Language Model Continual Learning (2023, background): “Existing continual learning works (Ke and Liu, 2022; Wang et al., 2023a) can be mainly categorized into rehearsal-based, regularization-based, and architecture-based approaches.”
• Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “In summary, this survey provides a comprehensive review of existing CL studies for LLMs, which significantly distinguishes itself from existing literature on related topics [17, 105, 237, 261, 276].”
• An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, unknown): “Scialom et al. (2022) [3] investigate the CF issues of T0 models during continual instruction tuning.”