Continual Learning with Lifelong Vision Transformer

By Zhen Wang, Liu Liu, Yiqun Duan, Yajing Kong, Dacheng Tao. 2022

99 citations recorded by the source corpus.

Continual learning methods aim at training a neural network from sequential data with streaming labels, relieving catastrophic forgetting. However, existing methods are based on and designed for convolutional neural networks (CNNs), which have not utilized the full potential of newly emerged powerful vision transformers. In this paper, we propose a novel attention-based framework Lifelong Vision Transformer (LVT), to achieve a better stability-plasticity trade-off for continual learning. Specifically, an inter-task attention mechanism is presented in LVT, which implicitly absorbs the previous tasks' information and slows down the drift of important attention between previous tasks and the current task. LVT designs a dual-classifier structure that independently injects new representation to avoid catas-trophic interference and accumulates the new and previous knowledge in a balanced manner to improve the overall performance. Moreover, we develop a confidence-aware memory update strategy to deepen the impression of the previous tasks. The extensive experimental results show that our approach achieves state-of-the-art performance with even fewer parameters on continual learning benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. René Vidal
6. Paris V. Giampouras
7. Liangzu Peng
8. Liu Liu

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jun Zhu
3. Hang Su
4. Liyuan Wang
5. Xingxing Zhang
6. René Vidal
7. Paris V. Giampouras
8. Liangzu Peng

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. René Vidal
6. Paris V. Giampouras
7. Liangzu Peng
8. Yajing Kong

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “This requires specialized designs to overcome catastrophic forgetting [439], [441] while providing new insights for maintaining task specificity in continual learning [438], [445], [446].”
• The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “…and more challenging case of task-agnostic or task-free continual learning, where in the training phase the learner has no access to task identities (Zeno et al., 2018; 2021; Aljundi et al., 2019a; Lee et al., 2020; Jin et al., 2021; Wang et al., 2022c; Pourcel et al., 2022; Ye & Bors, 2022).”
• Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, neutral): “Replayed-based methods leverage episodic memory to store representative history data or generate virtual data via a generative model, and replay these samples with current data [2, 7, 8, 12, 26, 37, 39, 40, 45, 50, 51].”
• Online Continual Learning with Contrastive Vision Transformer (2022, background): “Continual learning (also known as lifelong learning or incremental learning) [62,63,79,75,80,95,18] aims to solve this problem by maintaining and accumulating the acquired knowledge over time from a stream of non-stationary data.”
• Convolutional Prompting meets Language Models for Continual Learning (2024, methodology): “(3) Rehearsal-based methods [3, 4, 12, 20– 22, 41, 52] store a few representative training samples from previous tasks in a buffer, which is then used for training alongside the current task.”