Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay

By Yoojin Choi, Mostafa El-Khamy, Jungwon Lee. 2021

50 citations recorded by the source corpus.

This paper proposes two novel knowledge transfer techniques for class-incremental learning (CIL). First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples from a generative model. In the conventional generative replay, the generative model is pre-trained for old data and shared in extra memory for later incremental learning. In our proposed DF-GR, we train a generative model from scratch without using any training data, based on the pre-trained classification model from the past, so we curtail the cost of sharing pre-trained generative models. Second, we introduce dual-teacher information distillation (DT-ID) for knowledge distillation from two teachers to one student. In CIL, we use DT-ID to learn new classes incrementally based on the pre-trained model for old classes and another model (pre-)trained on the new data for new classes. We implemented the proposed schemes on top of one of the state-of-the-art CIL methods and showed the performance improvement on CIFAR-100 and ImageNet datasets.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Da-Wei Zhou
2. Han-Jia Ye
3. De-chuan Zhan
4. Qiwen Wang
5. Liang Zheng
6. Yi Yang
7. Qingfeng Lan
8. Yangchen Pan

Citations in the corpus

• Memory-Free Generative Replay For Class-Incremental Learning (2021, background): “To tackle this issue, [29] and [6] employ synthetic images that generates by data-free knowledge distillation methods [5, 32] to alleviate catastrophic forgetting.”
• DDGR: Continual Learning with Deep Diffusion-based Generative Replay (2023, unknown): “…11.13 23.19 19.73 54.60 54.57 58.12 59.89 46.65 49.31 47.70 49.62 DGR 42.49 38.16 52.96 48.94 43.94 38.81 53.32 47.56 24.08 26.52 26.36 31.14 17.31 22.52 17.96 Evaluation: In the CI scenario, we use two evaluation metrics which are similar to the work of Liu et al. (2020); Choi et al. (2021) .”
• A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “[46, 8] release the burden of exemplars by data-free knowledge distillation [33].”
• Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, methodology): “In generative replay methods, samples are stored in generative models rather than in a buffer (Shin et al. 2017, Kamra et al. 2017, Van de Ven and Tolias 2018, Ramapuram et al. 2020, Choi et al. 2021).”