Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners

By Duo Li, Guimei Cao, Yunlu Xu, Zhanzhan Cheng, Yi Niu. 2022

24 citations recorded by the source corpus.

In the SSLAD-Track 3B challenge on continual learning, we propose the method of COntinual Learning with Transformer (COLT). We find that transformers suffer less from catastrophic forgetting compared to convolutional neural network. The major principle of our method is to equip the transformer based feature extractor with old knowledge distillation and head expanding strategies to compete catastrophic forgetting. In this report, we first introduce the overall framework of continual learning for object detection. Then, we analyse the key elements' effect on withstanding catastrophic forgetting in our solution. Our method achieves 70.78 mAP on the SSLAD-Track 3B challenge test set.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Continual Learning in Vision Transformers: Recent work [6, 37, 69] has proven transformers to generalize well to unseen domains.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, background): “There have been also emerging some other transformer-based incremental learning methods like [71, 27, 37], and other incremental prompting methods like [42, 65].”
Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones (2022, positive): “[11] proposed a framework for object detection exploiting Swin Transformer [12] as pretrained backbone for a CascadeRCNN detector, the authors show that the extracted features generalize better to unseen domains hence achieving lesser forgetting rates compared to ResNet50 [8] backbones.”
Memory Efficient Continual Learning with Transformers (2022, methodology): “Among those recent works, only a few [30, 15] have applied the Transformers architecture to CL on image datasets.”
Continual Learning with Transformers for Image Classification (2022, methodology): “To the best of our knowledge, only a few recent works [8, 18] have applied the transformer architecture to CL on image datasets, but they also require rehearsal memory or training a new transformer model from scratch (cannot work with pre-trained transformer models).”
CoMFormer: Continual Learning in Semantic and Panoptic Segmentation (2022, methodology): “To the best of our knowledge, transformers have been evaluated only in continual learning for image classification [19, 31] and we are the first to use them in the continual segmentation task.”
Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization (2022, background): “It is worth noting that these works rely either on pretrained feature extractors [22] or rehearsal [15, 50] to defy forgetting.”