Online Continual Learning with Contrastive Vision Transformer
By Zhen Wang, Liu Liu, Yajing Kong, Jiaxian Guo, Dacheng Tao. 2022
45 citations recorded by the source corpus.
Online continual learning (online CL) studies the problem of learning sequential tasks from an online data stream without task boundaries, aiming to adapt to new data while alleviating catastrophic forgetting on the past tasks. This paper proposes a framework Contrastive Vision Transformer (CVT), which designs a focal contrastive learning strategy based on a transformer architecture, to achieve a better stability-plasticity trade-off for online CL. Specifically, we design a new external attention mechanism for online CL that implicitly captures previous tasks' information. Besides, CVT contains learnable focuses for each class, which could accumulate the knowledge of previous classes to alleviate forgetting. Based on the learnable focuses, we design a focal contrastive loss to rebalance contrastive learning between new and past classes and consolidate previously learned representations. Moreover, CVT contains a dual-classifier structure for decoupling learning current classes and balancing all observed classes. The extensive experimental results show that our approach achieves state-of-the-art performance with even fewer parameters on online CL benchmarks and effectively alleviates the catastrophic forgetting.
Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Radiology, Nuclear Medicine and Imaging
In Representation & Self-Supervised Learning
In Meta & Continual Learning
Citations in the corpus
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “This requires specialized designs to overcome catastrophic forgetting [439], [441] while providing new insights for maintaining task specificity in continual learning [438], [445], [446].”
- Deep Class-Incremental Learning: A Survey (2023, background): “Future class-incremental learning algorithms should be able to conduct fully online training [226], [227], [228], [229], [230] without requiring the task boundaries [231], [232], [233].”