R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning

By Qiankun Gao, Chen Zhao, Bernard Ghanem, Jian Zhang. 2022

102 citations recorded by the source corpus.

Class-Incremental Learning (CIL) struggles with catastrophic forgetting when learning new knowledge, and Data-Free CIL (DFCIL) is even more challenging without access to the training data of previously learned classes. Though recent DFCIL works introduce techniques such as model inversion to synthesize data for previous classes, they fail to overcome forgetting due to the severe domain gap between the synthetic and real data. To address this issue, this paper proposes relation-guided representation learning (RRL) for DFCIL, dubbed R-DFCIL. In RRL, we introduce relational knowledge distillation to flexibly transfer the structural relation of new data from the old model to the current model. Our RRL-boosted DFCIL can guide the current model to learn representations of new classes better compatible with representations of previous classes, which greatly reduces forgetting while improving plasticity. To avoid the mutual interference between representation and classifier learning, we employ local rather than global classification loss during RRL. After RRL, the classification head is refined with global class-balanced classification loss to address the data imbalance issue as well as learn the decision boundaries between new and previous classes. Extensive experiments on CIFAR100, Tiny-ImageNet200, and ImageNet100 demonstrate that our R-DFCIL significantly surpasses previous approaches and achieves a new state-of-the-art performance for DFCIL. Code is available at https://github.com/jianzhangcs/R-DFCIL

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Da-Wei Zhou
2. Han-Jia Ye
3. De-chuan Zhan
4. Qiwen Wang
5. Ziwei Liu
6. Rameswar Panda
7. Hai-Long Sun
8. James Smith

Citations in the corpus

• CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Rehearsal-Free Continual Learning: Recent works propose producing images for rehearsal using deep-model inversion [12, 17, 56, 68].”
• Deep Class-Incremental Learning: A Survey (2023, background): “R-DFCIL [101] suggests mapping the angle among triplets:∑ {xi,xj ,xk}∈Db ‖cos∠titjtk − cos∠sisjsk‖ , (16) where tm = φb−1(xm) is the representation in the old model’s embedding space, and sm = φ(xm) denotes the representation in the current model.”
• Class-Incremental Learning: A Survey (2023, background): “R-DFCIL [99] suggests mapping the angle among triplets: where t m = ϕ b − 1 ( x m ) is the representation in the old model’s embedding space, and s m = ϕ ( x m ) denotes the representation in the current model.”
• Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning (2024, background): “Class-Incremental Learning (CIL) : requires a learning system to continually absorb new class knowledge without forgetting existing ones [13, 14, 20, 22, 38, 57, 59, 74, 81, 82], which can be roughly divided into several categories.”
• Consistent Prompting for Rehearsal-Free Continual Learning (2024, unknown): “Therefore, rehearsal-free continual learning meth-ods [9, 23, 25, 35, 43, 51] have attracted much attention.”
• A Unified Continual Learning Framework with General Parameter-Efficient Tuning (2023, unknown): “…do not rely on old samples that may raise privacy concerns, they dynamically expand the network or isolate parameters for different tasks [41, 50, 26, 18], regularize the network parameters that are important to learned tasks [28, 20, 52, 1], and replay generative or synthetic data [42, 49, 43, 8].”