Lifelong Language Knowledge Distillation

By Yung-Sung Chuang, Shang-Yu Su, Yun-Nung Chen. 2020

57 citations recorded by the source corpus.

It is challenging to perform lifelong language learning (LLL) on a stream of different tasks without any performance degradation comparing to the multi-task counterparts. To address this issue, we present Lifelong Language Knowledge Distillation (L2KD), a simple but efficient method that can be easily applied to existing LLL architectures in order to mitigate the degradation. Specifically, when the LLL model is trained on a new task, we assign a teacher model to first learn the new task, and pass the knowledge to the LLL model via knowledge distillation. Therefore, the LLL model can better adapt to the new task while keeping the previously learned knowledge. Experiments show that the proposed L2KD consistently improves previous state-of-the-art models, and the degradation comparing to multi-task models in LLL tasks is well mitigated for both sequence generation and text classification tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zixuan Ke
2. Bing Liu
3. Hu Xu
4. Lei Shu
5. Bin Liu
6. Gyuhak Kim
7. Tatsuya Konishi
8. Nianzu Ma

In Computer Vision and Pattern Recognition

1. Zixuan Ke
2. Bin Liu
3. Hu Xu
4. Bing Liu
5. Gyuhak Kim
6. B. Ermiş
7. Giovanni Zappella
8. Martin Wistuba

In Natural Language Processing

1. Hu Xu
2. Lei Shu
3. Bing Liu
4. Zixuan Ke
5. Xuezhi Wang
6. Xiang Ren
7. Yanzhe Zhang
8. Nianzu Ma

In Meta & Continual Learning

1. Zixuan Ke
2. Bing Liu
3. Bin Liu
4. Gyuhak Kim
5. Hu Xu
6. Lei Shu
7. Tatsuya Konishi
8. Nianzu Ma

Citations in the corpus

• Modifying Memories in Transformer Models (2020, background): “Continual learning, with recent extensions to language models (Sun et al., 2020; Liu et al., 2019; Mi et al., 2020; Chuang et al., 2020), aims to learn a new task while preserving the performance on the previous tasks without access to their data.”
• Continual Training of Language Models for Few-Shot Learning (2022, background): “…(Liu et al., 2019a), translation (Khayral-lah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classification (Ke et al., 2021a,b; Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020; Mehta et al., 2021; Madotto et al., 2020).”
• Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection (2022, background): “, 2022) and knowledge distillation (Chuang et al., 2020; Cao et al., 2020) have shown to be effective in overcoming catastrophic forgetting, they highly rely on the stored data from old tasks, which is not the most realistic setting for lifelong learning.”
• Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “The first one continually learns new tasks on old parameters (Fig 1 a), with approaches like experience replay (Sun et al., 2019; Chuang et al., 2020) and regularization (Mi et al.”
• Memory Efficient Continual Learning with Transformers (2022, background): “Recent work studied catastrophic forgetting [55, 10, 27, 35] and incremental learning [66] for NLP and CV [32, 15].”
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “For example, CL models have been proposed for sentiment analysis [23, 24, 37, 43], dialogue slot filling [53], language modeling [58, 7], language learning [31], sentence embedding [33], machine translation [25], cross-lingual modeling [35], and question answering [12].”
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, background): “These include LAMOL (Sun et al., 2019) and its improvements (Chuang et al., 2020; Sun et al., 2020; Kanwatchara et al., 2021).”
• Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, methodology): “There have been recent extensions of the algorithm, such as L2KD (Chuang et al., 2020), which performs knowledge distilla-”
• Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue (2022, methodology): “Following recent studies (Sun et al., 2020; Chuang et al., 2020), PCLL unifies different NLU tasks into a language modeling (LM) task and implements a task solver based on a PLM.”
• MeLL: Large-scale Extensible User Intent Classification for Dialogue Systems with Meta Lifelong Learning (2021, methodology): “Typical methods include experience replay [16, 20, 40], knowledge distillation [6, 26], transfer learning [5, 19, 39], etc.”
• Memory Efficient Continual Learning for Neural Text Classification (2022, background): “Recent work studied catastrophic forgetting (Sun et al., 2019; Chuang et al., 2020; Ke et al., 2020; Liu et al., 2020) and incremental learning (Xia et al., 2021) for NLP. Pa-sunuru et al. (2021) focus on the few-shot setting where only a few data points are available for each task.”
• Continual Pre-training of Language Models (2023, methodology): “In NLP, CL has been used for slot filling (Shen et al., 2019), language learning (Li et al., 2019), sentiment analysis (Ke et al., 2021a), topic modeling (Gupta et al., 2020), question answering (Greco et al., 2019) and text classification (Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020).”