Memory Efficient Continual Learning with Transformers

By B. Ermiş, Giovanni Zappella, Martin Wistuba, Aditya Rawal, Cédric Archambeau. 2022

70 citations recorded by the source corpus.

In many real-world scenarios, data to train machine learning models becomes available over time. Unfortunately, these models struggle to continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is known as catastrophic forgetting and it is difficult to prevent due to practical constraints. For instance, the amount of data that can be stored or the computational resources that can be used might be limited. Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since the resources or data might not be available in sufficiently large quantities to practitioners to train the model from scratch. In this paper, we devise a method to incrementally train a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters. Different than the existing approaches, our method is able to scale to a large number of tasks without significant overhead and allows sharing information across tasks. On both image and text classification tasks, we empirically demonstrate that our method maintains a good predictive performance without retraining the model or increasing the number of model parameters over time. The resulting model is also significantly faster at inference time compared to Adapter-based state-of-the-art methods.

Subfields: Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

1. Liyuan Wang
2. Peter Stone
3. Jun Zhu
4. Bo Liu
5. De-chuan Zhan
6. Hang Su
7. Da-Wei Zhou
8. Ziwei Liu

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Xingxing Zhang
5. Liyuan Wang
6. Hang Su
7. Qiwen Wang
8. Jun Zhu

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Representative strategies include prompting [402], [457], [458] or carefully fine-tuning [303], [389], [504] the backbone, saving prototypes to rectify the predictions [188], [335], [504], incorporating transfer learning techniques [43], [114], [330], etc.”
• Deep Class-Incremental Learning: A Survey (2023, background): “On the other hand, several works [153], [154], [155] propose to design specific modules for incremental new tasks, e.”
• Class-Incremental Learning: A Survey (2023, background): “As a result, algorithms should be designed to address catastrophic forgetting in two aspects ( i.e. , classifier aspect and generative model aspect) when using generative replay.”
• Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, background): “We notice that the recent works [47], [48] also explored the mechanisms of selecting the most suitable sub-model for the coming tasks.”
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (2023, background): “[21] Beyza Ermis, Giovanni Zappella, Martin Wistuba, and Cédric Archambeau.”