Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay

By Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, Jingshan Tang. 2022

137 citations recorded by the source corpus.

Few-shot class-incremental learning (FSCIL) has been proposed aiming to enable a deep learning system to incrementally learn new classes with limited data. Recently, a pioneer claims that the commonly used replay-based method in class-incremental learning (CIL) is ineffective and thus not preferred for FSCIL. This has, if truth, a significant influence on the fields of FSCIL. In this paper, we show through empirical results that adopting the data replay is surprisingly favorable. However, storing and replaying old data can lead to a privacy concern. To address this issue, we alternatively propose using data-free replay that can synthesize data by a generator without accessing real data. In observing the the effectiveness of uncertain data for knowledge distillation, we impose entropy regularization in the generator training to encourage more uncertain examples. Moreover, we propose to relabel the generated data with one-hot-like labels. This modification allows the network to learn by solely minimizing the cross-entropy loss, which mitigates the problem of balancing different objectives in the conventional knowledge distillation approach. Finally, we show extensive experimental results and analysis on CIFAR-100, miniImageNet and CUB-200 to demonstrate the effectiveness of our proposed one.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xin Ning
2. Zhenyu Weng
3. Ziqian Zeng
4. Weijun Li
5. Prayag Tiwari
6. Zhiping Lin
7. Hang Ran
8. Run He

In Meta & Continual Learning

1. Zhenyu Weng
2. Ziqian Zeng
3. Zhiping Lin
4. Run He
5. Huiping Zhuang

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “…graph for knowledge distillation [250], selectively updating only the unimportant parameters [251] or stabilizing the important parameters [252], updating parameters within the flat region of loss landscape [84], meta-learning of a good initialization [253], as well as generative replay [254].”
• A Survey on Few-Shot Class-Incremental Learning (2023, background): “In the majority of FSCIL experiments (Cheraghian et al., 2021c; Chi et al., 2022; Liu et al., 2022b; Peng et al., 2022; SHI et al., 2021; Tao et al., 2020; Yang et al., 2022; Zhang et al., 2021; Zheng and Zhang, 2021; Zhou et al., 2022b; Zhu et al., 2021a), the three image classification datasets…”
• Consistent Prompting for Rehearsal-Free Continual Learning (2024, unknown): “Therefore, rehearsal-free continual learning meth-ods [9, 23, 25, 35, 43, 51] have attracted much attention.”
• GKEAL: Gaussian Kernel Embedded Analytic Learning for Few-Shot Class Incremental Task (2023, background): “For the most recent FSCIL teniques like MetaFSCIL and Entropy-reg, they slightly outperforms CEC and F2M.”