FOSTER: Feature Boosting and Compression for Class-Incremental Learning

By Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, De-chuan Zhan. 2022

428 citations recorded by the source corpus.

The ability to learn new concepts continually is necessary in this ever-changing world. However, deep neural networks suffer from catastrophic forgetting when learning new categories. Many works have been proposed to alleviate this phenomenon, whereas most of them either fall into the stability-plasticity dilemma or take too much computation or storage overhead. Inspired by the gradient boosting algorithm to gradually fit the residuals between the target model and the previous ensemble model, we propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively. Specifically, we first dynamically expand new modules to fit the residuals between the target and the output of the original model. Next, we remove redundant parameters and feature dimensions through an effective distillation strategy to maintain the single backbone model. We validate our method FOSTER on CIFAR-100 and ImageNet-100/1000 under different settings. Experimental results show that our method achieves state-of-the-art performance. Code is available at: https://github.com/G-U-N/ECCV22-FOSTER.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “FOSTER [155] expands new modules to ﬁt the residuals of the old model and then distills them into a single model.”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, background): “Despite the same rebounding effect in some methods, e.g., FOSTER, they fail to remain close to NNCSL.”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “Class-Incremental Learning is now a popular topic in the machine learning ﬁeld [25], [26], [27], [28], [29], which can be roughly divided into two groups by whether saving exem-plars of old class instances.”
TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation (2023, methodology): “In CL, exemplar-based methods [44, 33, 54, 13] have achieved leading performance.”
Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning (2024, methodology): “Recently, expandable networks [10, 17, 29, 30, 56, 64] show strong performance among other competitors.”
Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning (2024, methodology): “Exemplar-based CIL methods [2, 3, 6, 8, 13, 25, 38, 49] store small subsets of data from each task.”
A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “Rather than storing exemplars, recent works [54, 50, 30, 12] find that saving backbones from the history pushes the performance by one step towards the upper bound.”
Deep Class-Incremental Learning: A Survey (2023, background): “However, recent methods based on dynamic networks (i.e., DER, FOSTER, MEMO, and DyTox) show competitive results with the help of multiple backbones.”
Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, positive): “In addition to distilla-3849 tion methods appearing in continual segmentation, dynamic networks with parameter expansion [13, 37, 41, 41, 45, 49, 50, 54, 64] also play a key role in continual learning.”
Divide and not forget: Ensemble of selectively trained experts in Continual Learning (2024, background): “The simplest solution to fight catastrophic forgetting in CIL is to store exemplars, e.g. LUCIR (Hou et al., 2019), BiC (Wu et al., 2019), Foster (Wang et al., 2022a), WA (Zhao et al., 2020).”
Continual Detection Transformer for Incremental Object Detection (2023, neutral): “Recent incremental learning methods can be divided into two categories: (i) Knowledge Distillation (KD) tries to preserve the knowledge capture in a previous version of the model by matching logits [27,43], feature maps [12], or other information [19,32,41,47,50,53] in the new model.”
Class-Incremental Learning: A Survey (2023, background): “FOSTER [16] formulates the learning process in Eq.”
Consistent Prompting for Rehearsal-Free Continual Learning (2024, positive): “The aforementioned paradigms are highly complementary and can be combined to enhance continual learning performance [5, 40, 41, 48].”