Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks

By Zixuan Ke, Hu Xu, Bing Liu. 2021

95 citations recorded by the source corpus.

This paper studies continual learning (CL) of a sequence of aspect sentiment classification (ASC) tasks. Although some CL techniques have been proposed for document sentiment classification, we are not aware of any CL work on ASC. A CL system that incrementally learns a sequence of ASC tasks should address the following two issues: (1) transfer knowledge learned from previous tasks to the new task to help it learn a better model, and (2) maintain the performance of the models for previous tasks so that they are not forgotten. This paper proposes a novel capsule network based model called B-CL to address these issues. B-CL markedly improves the ASC performance on both the new task and the old tasks via forward and backward knowledge transfer. The effectiveness of B-CL is demonstrated through extensive experiments.

Subfields: Artificial Intelligence, Large Language Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zixuan Ke
2. Bing Liu
3. Han-Jia Ye
4. De-chuan Zhan
5. Da-Wei Zhou
6. Hu Xu
7. Lei Shu
8. Bin Liu

In Large Language Models

1. Zixuan Ke
2. Bin Liu
3. Hu Xu
4. Lei Shu
5. Bing Liu
6. Thomas Scialom
7. Tuhin Chakrabarty
8. Smaranda Muresan

In Representation & Self-Supervised Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Zixuan Ke
4. Da-Wei Zhou
5. Hu Xu
6. Lei Shu
7. Smaranda Muresan
8. Tuhin Chakrabarty

Citations in the corpus

• RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, background): “While there are ways to alleviate this (Ke et al., 2021), this is beyond the scope of this paper.”
• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “For example, a pre-trained transformer can effectively accommodate downstream tasks by learning only a few task-specific parameters, including adaptor-tuning [165] with inserted fully-connected layers (CTR [208], B-CL [209], CLASSIC [207], CPT [205], CLIF [190], AdapterCL [282], ACM [499], ADA [111]”
• Continual Training of Language Models for Few-Shot Learning (2022, background): “(10) BCL (Ke et al., 2021c) is a continual learning model that can avoid forgetting and encourage knowledge transfer.”
• Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection (2022, background): “However, when moving to lifelong learning setting, their performance significantly drops (Kirkpatrick et al., 2017; Aljundi et al., 2019; Biesialska et al., 2020; Cui et al., 2021; Ke et al., 2021b; Madotto et al., 2021; Ke et al., 2021a; Feng et al., 2022).”
• Continual Sequence Generation with Adaptive Compositional Modules (2022, background): “Extra modules like attention module (Pfeiffer et al., 2020a), capsule network (Ke et al., 2021), and hypernetworks (Jin et al., 2021) are demonstrated beneficial for knowledge transfer, but they need to introduce extra parameters and fail to consider any reusable or compositional modules.”
• ConTinTin: Continual Learning from Task Instructions (2022, neutral): “In addi-Method forward-transfer backward-transfer Multi-task (upperbound) 7.98 ± 20.47 et al., 2019; Jin et al., 2021; Ke et al., 2021).”
• Memory Efficient Continual Learning with Transformers (2022, background): “[28] proposed an architecture to achieve both CF prevention and knowledge transfer.”
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, methodology): “But such pre-trained models or feature extractors can significantly improve the CL performance [18, 24].”
• Few-Shot Class-Incremental Learning for Named Entity Recognition (2022, background): “Sun et al. (2019); Ke et al. (2021) have study continual learning for different tasks of NLP. Recently, Monaikul et al. (2021) studies class-incremental learning for NER, building a unified NER classifier for all the classes encountered over time.”
• CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks (2021, background): “, the data owners do not want their data used by others (Ke et al., 2020b; Qin et al., 2020; Ke et al., 2021).”
• Fine-tuned Language Models are Continual Learners (2022, methodology): “Cao et al. (2021) propose a new CL framework for NMT models, while Ke et al. (2021) proposes a novel capsule network based model called B-CL (Bert based CL) for sentiment classification tasks.”
• Memory Efficient Continual Learning for Neural Text Classification (2022, methodology): “Ke et al. (2021) proposed an architecture to achieve both CF prevention and knowledge transfer.”
• A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, methodology): “, 2018) for ASC dataset (Ke et al., 2021) (the aspect sentiment classification dataset containing 19 products), which is a benchmark NLP task.”
• Continual Few-shot Intent Detection (2022, background): “However, there are two issues: (i) over-parameterization of PrLMs makes them prone to overfit the current task and cause catastrophic forgetting of previous knowledge (Ke et al., 2021a; Yuan et al., 2021).”
• Continual Pre-training of Language Models (2023, methodology): “Those baselines (BCL, CLASSIC and DEMIX) perform knowledge transfer achieve better results but still poorer than DAS.”