Continual Learning for Text Classification with Information Disentanglement Based Regularization

By Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, Diyi Yang. 2021

141 citations recorded by the source corpus.

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The applications of continual learning in NLP are also highly diverse, such as dialogue system [130], [131], [260], [260], [282], [298], [437], text classification [87], [172], [325], [460], sentence generation [246], [298], [499], relation learning [343], [357], [455], [465], neural machine transla”
Progressive Prompts: Continual Learning for Language Models (2023, methodology): “• IDBR (Huang et al., 2021): BERT-specific approach which continuously trains the whole model using data replay and a regularization loss, which applies sentence representation disentanglement into task-specific and task-generic spaces.”
Continual Training of Language Models for Few-Shot Learning (2022, background): “…(Liu et al., 2019a), translation (Khayral-lah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classiﬁcation (Ke et al., 2021a,b; Sun et al., 2020; Huang et al., 2021; Chuang et al., 2020; Mehta et al., 2021; Madotto et al., 2020).”
Meta-attention for ViT-backed Continual Learning (2022, methodology): “[17] presents an information disentanglement-based regularization method to further generalize old task knowledge.”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “Comparing to continual learning on text classiﬁ-cation and question answering (Wang et al., 2020; Holla et al., 2020; Huang et al., 2021), continual sequence generation is more challenging, since the output is no longer discrete labels but sequential text data in different styles/domains.”
Memory Efficient Continual Learning with Transformers (2022, methodology): “[25] Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang.”
Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning (2021, background): “Recently, continual learning has drawn attention in the NLP field (Sun et al., 2020; Wang et al., 2019b; Huang et al., 2021).”
Memory Efficient Continual Learning for Neural Text Classification (2022, methodology): “We observe TransRate performing generally better than LEEP, as in previous experiments.”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “, 2021) or learning disparate tasks (de Masson d'Autume et al., 2019; Huang et al., 2021).”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “Other lines of works focus on regularization over the sentence representations (Wang et al., 2019; Huang et al., 2021; Liu et al., 2019a) or directly merging models in the parameter space (Matena and Raffel, 2021).”