Prototype Augmentation and Self-Supervision for Incremental Learning

By Fei Zhu, Xu-Yao Zhang, Chuan Wang, Fei Yin, Chenglin Liu. 2021

529 citations recorded by the source corpus.

Despite the impressive performance in many individual tasks, deep neural networks suffer from catastrophic forgetting when learning new tasks incrementally. Recently, various incremental learning methods have been proposed, and some approaches achieved acceptable performance relying on stored data or complex generative models. However, storing data from previous tasks is limited by memory or privacy issues, and generative models are usually unstable and inefficient in training. In this paper, we propose a simple non-exemplar based method named PASS, to address the catastrophic forgetting problem in incremental learning. On the one hand, we propose to memorize one class-representative prototype for each old class and adopt prototype augmentation (protoAug) in the deep feature space to maintain the decision boundary of previous tasks. On the other hand, we employ self-supervised learning (SSL) to learn more generalizable and transferable features for other tasks, which demonstrates the effectiveness of SSL in incremental learning. Experimental results on benchmark datasets show that our approach significantly outperforms non-exemplar based methods, and achieves comparable performance compared to exemplar based approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Feature Space LUCIR [164], PODNet [100], TPCIL [403], PCL [170], AANets [270], DER [468], DDE [171], GeoDL [387], PASS [510], Co2L [57], AFC [199], SP-CIL [456], ELI [192], CwD [380], CSCCT [22], FOSTER [424], FASP [300], CLS-ER [18]”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “This has similarity with feature augmentations which are also commonly used [47].”
Learnability and Algorithm for Continual Learning (2023, methodology): “The exemplarfree methods are: HAT (Serrà et al., 2018), OWM (Zeng 2https://github.com/k-gyuhak/CLOOD et al., 2019), SLDA (Hayes & Kanan, 2020), PASS (Zhu et al., 2021), and L2P (Wang et al., 2022c).”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “Please note that our CA or CA+LN is operated in a post-hoc fashion rather than aligning the classifier during representation learning [56, 11, 12], as the latter would even worsen the performance (e.”
Towards Label-Efficient Incremental Learning: A Survey (2023, methodology): “, 2021] Self-Supervised IL Auxiliary-training Self & Label PASS [Zhu et al., 2021] Self-Supervised IL Main-training Self-only CaSSLe [Fini et al.”
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, methodology): “FeTrIL shares the idea of using class prototypes with [45, 47, 48, 49].”
A Theoretical Study on Solving Continual Learning (2022, methodology): “For the backbone structure, we follow [4, 26, 34].”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “For the proxy-based continual learning, we follow the same experimental setting as in (Zhu et al., 2021).”
General Incremental Learning with Domain-aware Categorical Representations (2022, background): “It is also worth noting that while continual learning without data memory has attracted much attention in literature [32, 40, 42], these methods typically perform less effectively than those with a limited data memory for storing old examples [9, 37].”
Forward Compatible Few-Shot Class-Incremental Learning (2022, methodology): “Other methods rehearsal former instances to overcome forgetting [6, 48, 60, 66].”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, methodology): “To better assess the overall performance of our scheme, we compare it to the SOTA of NECIL (EWC∗, LwF_MC∗, MUC∗, SDC and PASS) and some classical methods of exemplar-based CIL (iCARL∗, EEIL∗ and UCIR∗).”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “The knowledge can be stored by multiple types, like examples [4, 6, 9, 11, 73, 83], prototypes [35, 106, 107], generative networks [60], etc.”
Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “Even if in image classification it has been exhaustively studied [1, 11, 20, 22, 24, 30, 38, 41, 54, 57, 68], in semantic segmentation it is still in its early stages [8, 9, 21, 33, 43, 45–47].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “[115] proposed to store a class-representative prototype for each previous task and augment these prototypes to preserve the decision boundaries of the previous tasks.”
Co-Transport for Class-Incremental Learning (2021, background): “Class-incremental learning (CIL) is now a popular topic in the machine learning community [6, 9, 15, 25, 39, 47, 62, 68, 70].”
Class-Incremental Learning: A Survey (2023, background): “[49], [120] draw inspiration from contrastive learning and design pre-text tasks to enhance the representation learned by the first stage.”
Open-World Continual Learning: Unifying Novelty Detection and Continual Learning (2023, methodology): “The values τk and k are hyper-parameters and we use the same values for all tasks except for PASS, for which we use a validation set to tune τk (see Appendix B).”
Bring Evanescent Representations to Life in Lifelong Class Incremental Learning (2022, methodology): “We use a per-step incremental accuracy metric [43] deﬁned as the average top-1 classiﬁcation accuracy over all classes observed up to the current incre-Table [9] in fusion loss.”