Improving Vision Transformers for Incremental Learning

By Pei Yu, Yinpeng Chen, Ying Jin, Zicheng Liu. 2021

18 citations recorded by the source corpus.

This paper proposes a working recipe of using Vision Transformer (ViT) in class incremental learning. Although this recipe only combines existing techniques, developing the combination is not trivial. Firstly, naive application of ViT to replace convolutional neural networks (CNNs) in incremental learning results in serious performance degradation. Secondly, we nail down three issues of naively using ViT: (a) ViT has very slow convergence when the number of classes is small, (b) more bias towards new classes is observed in ViT than CNN-based architectures, and (c) the conventional learning rate of ViT is too low to learn a good classifier layer. Finally, our solution, named ViTIL (ViT for Incremental Learning) achieves new state-of-the-art on both CIFAR and ImageNet datasets for all three class incremental learning setups by a clear margin. We believe this advances the knowledge of transformer in the incremental learning community. Code will be publicly released.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Liu Liu
3. Zhen Wang
4. Yajing Kong
5. Rameswar Panda
6. Rogério Schmidt Feris
7. James Smith
8. Joost van de Weijer

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Rogério Schmidt Feris
4. Ming-Hsuan Yang
5. Xiaogang Wang
6. Rameswar Panda
7. Andrea Vedaldi
8. Cordelia Schmid

In Radiology, Nuclear Medicine and Imaging

1. Yajing Kong
2. Dacheng Tao
3. Zsolt Kira
4. Zhiwu Huang
5. Rogério Schmidt Feris
6. Rameswar Panda
7. Liu Liu
8. Yabin Wang

In Meta & Continual Learning

1. Dacheng Tao
2. James Smith
3. Liu Liu
4. Zhen Wang
5. Joost van de Weijer
6. Zsolt Kira
7. Yajing Kong
8. Rameswar Panda

Citations in the corpus

• CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Another [70] shows that the vanilla counterparts of ViTs are more prone to forgetting when trained from scratch.”
• S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, background): “There have been also emerging some other transformer-based incremental learning methods like [71, 27, 37], and other incremental prompting methods like [42, 65].”
• Online Continual Learning with Contrastive Vision Transformer (2022, background): “Since then, a lot of effort has been dedicated to improving Vision Transformers’ data efficiency and model efficiency [92,32,91,40], where an effective direction is to strategically integrate properties of convolution into the Transformer architecture [73,91,48,84,26,94,23,17].”
• Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones (2022, methodology): “Several methods in CL use pretrained backbones as feature extractors such as in Hayes et al [7] or [1,9] and some-times the pretraining is carried on half (or a big portion) the dataset considered, as in PODNet [4] or in Yu et al. [26].”
• Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization (2022, background): “Convolutional stems have previously been shown to improve performance and convergence speed in incremental learning settings [50].”