Continual-T0: Progressively Instructing 50+ Tasks to Language Models Without Forgetting

By Thomas Scialom, Tuhin Chakrabarty, Smaranda Muresan. 2022

22 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Health Informatics, Large Language Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Mohit Bansal
3. Graham Neubig
4. Yejin Choi
5. Jianfeng Gao
6. Percy Liang
7. Hannaneh Hajishirzi
8. Jason Aaron Edward Weston

In Health Informatics

1. Bin Liu
2. Zixuan Ke

In Large Language Models

1. Luke S. Zettlemoyer
2. Timo Schick
3. Kurt Shuster
4. Bin Liu
5. Thomas Scialom
6. Xian Li
7. Edouard Grave
8. Asli Celikyilmaz

In Meta & Continual Learning

1. Bin Liu
2. Zixuan Ke
3. Ramakanth Pasunuru
4. Yann LeCun
5. Timo Schick
6. Kurt Shuster
7. Bing Liu
8. Gyuhak Kim

Citations in the corpus

• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “, 2022) or examples (Scialom et al., 2022) of previously learned tasks when fine-tuning with a new task to prevent catastrophic forgetting.”
• Augmented Language Models: a Survey (2023, background): “, 2022), the size and need for data of LLMs can be impractical for training but also maintenance: continual learning for large models remains an open research question (Scialom et al., 2022).”
• Galactica: A Large Language Model for Science (2022, background): “The question of how to update network knowledge remains an active research question (Scialom et al., 2022; Mitchell et al., 2022).”
• Scaling Instruction-Finetuned Language Models (2022, neutral): “This finding is consistent with prior studies that demonstrated the effectiveness of instruction finetuning on either T5 models (Sanh et al., 2021; Wang et al., 2022c; Scialom et al., 2022) or decoder-only language models (Wei et al., 2021; Ouyang et al., 2022).”
• Continual Learning of Natural Language Processing Tasks: A Survey (2022, methodology): “…summarization and information extraction), which is usually supervised [Ke et al., 2021a; Monaikul et al., 2021; Qin and Joty, 2022], or a corpus domain used to further pre-train or post-train an LM, which is usually unsupervised [Gururangan et al., 2021; Ke et al., 2022; Scialom et al., 2022].”
• Emergent and Predictable Memorization in Large Language Models (2023, background): “This can be understood as a form of continual learning in large language models [48] via implicit experience replay [38].”