LAMOL: LAnguage MOdeling for Lifelong Language Learning

By Fan-Keng Sun, Cheng-Hao Ho, Hung-yi Lee. 2019

253 citations recorded by the source corpus.

Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudo-samples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Meta & Continual Learning

Citations in the corpus

Modifying Memories in Transformer Models (2020, background): “ls can improve their generalization by boosting their factual knowledge in speciﬁc domains. 3 Memory modiﬁcation vs. continual learning. Continual learning, with recent extensions to language models (Sun et al., 2020; Liu et al., 2019; Mi et al., 2020; Chuang et al., 2020), aims to learn a new task ”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “MBPA++ [442], EMAR [465], DnR [396], ARPER [298], Total Recall [253]), generative replay (PAGeR [415], LAMOL [395], ACM [499], NER [434]), parameter allocation (TPEM [131]), modular network (ProgModel [377]), meta-learning (MetaMBPA++ [442], MeLL [422], CML [455]), etc.”
Progressive Prompts: Continual Learning for Language Models (2023, methodology): “These methods use data replay or add regularization constraints (Huang et al., 2021; de Masson D’Autume et al., 2019; Sun et al., 2019), but they still suffer from forgetting due to inevitable changes in parameters shared between tasks.”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “In such cases of data distributions shift, current generation models often suffer from catastrophic forgetting (Sun et al., 2019): models completely and abruptly forget previously learned information upon learning new information.”
ConTinTin: Continual Learning from Task Instructions (2022, background): “Systems in (Sun et al., 2020a,b) incrementally learned among ﬁve disparate NLP tasks.”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, neutral): “…2017; Rolnick et al., 2019), which perform experience replay with authentic data (de Masson d’Autume et al., 2019), automatically generated data (Sun et al., 2020), or previously computed gradients (Lopez-Paz and Ranzato, 2017) conserved in the memory, (2) consolidation-based methods…”
Memory Efficient Continual Learning with Transformers (2022, background): “Recent work studied catastrophic forgetting [55, 10, 27, 35] and incremental learning [66] for NLP and CV [32, 15].”
Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation (2022, background): “In 839 Proceedings of the 4th Workshop on Representation 840 Learning for NLP (RepL4NLP-2019), pages 224– 841 229, Florence, Italy.”
LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, background): “Recently, Sun et al. (2019) proposes LAMOL, a general framework designed for lifelong language learning (LLL), where the model needs to continually learn from different domains as well as different types of NLP tasks.”
Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021, background): “…learning is applied to tasks such as relation extraction (Wang et al., 2019; Han et al., 2020), natural language generation (Mi et al., 2020), language modelling (Sun et al., 2019) and the pretrained language models adapting to multiple NLP tasks (Wang et al., 2020b; Pfeiffer et al., 2021).”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, background): “[45] Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee.”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, methodology): “• LAMOL (Sun et al., 2019): train a language model that simultaneously learns to solve the tasks and generate training samples, the latter”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Moreover, multiple works are proposed to address general CL language learning [138, 139] and specific language tasks, such as dialogue systems [140, 141, 142], image captioning [143], sentiment classification [144] and sentence representation learning [145].”
Distill and Replay for Continual Language Learning (2020, methodology): “1https://huggingface.co/gpt2 • Data-based methods: LAMOL2, the previous state-of-the-art continual language learning method, is adopted.”