Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning

By James Smith, Yen-Chang Hsu, Jonathan C. Balloch, Yilin Shen, Hongxia Jin, Zsolt Kira. 2021

213 citations recorded by the source corpus.

Modern computer vision applications suffer from catastrophic forgetting when incrementally learning new concepts over time. The most successful approaches to alleviate this forgetting require extensive replay of previously seen data, which is problematic when memory constraints or data legality concerns exist. In this work, we consider the high-impact problem of Data-Free Class-Incremental Learning (DFCIL), where an incremental learning agent must learn new concepts over time without storing generators or training data from past tasks. One approach for DFCIL is to replay synthetic images produced by inverting a frozen copy of the learner’s classification model, but we show this approach fails for common class-incremental benchmarks when using standard distillation strategies. We diagnose the cause of this failure and propose a novel incremental distillation strategy for DFCIL, contributing a modified cross-entropy training and importance-weighted feature distillation, and show that our method results in up to a 25.1% increase in final task accuracy (absolute difference) compared to SOTA DFCIL methods for common class-incremental benchmarks. Our method even outperforms several standard replay based methods which store a coreset of images. Our code is available at https://github.com/GT-RIPL/AlwaysBeDreaming-DFCIL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “An intriguing idea is to replay synthetic data produced by inverting a frozen copy of the old classiﬁcation model [247], [248], which usually further incorporate knowledge distillation to compensate the lost information in model inversion.”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “In this paper, we instead consider the highly-impactful and well-established setting of rehearsal-free continual learning [39, 57, 58, 66, 67] 1 , limiting our scope to strategies for continual learning which do not store training data.”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “Due to the distribution shift between synthetic and real examples, the model is frequently biased if regarding them as positive examples [42].”
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, background): “Recently, [26,30,29] have integrated the idea of generative reply into addressing class-incremental learning problem using Deep-Inversion.”
Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, methodology): “Existing data-driven methods [3, 4, 6, 11, 16, 18, 22, 27, 37] focus on the data and the relation of new and old data to tackle forgetting.”
Continual Sequence Generation with Adaptive Compositional Modules (2022, background): “…follows a line of work that doesn’t store real examples for experience replay, such as generating examples by GAN (Atkinson et al., 2018), synthesizing examples (Xu et al., 2022) by model-inversion (Smith et al., 2021b), and using unlabeled data in the learning environment (Smith et al., 2021a).”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, methodology): “B0 setting is adopted in [9], where all 100 classes are divided into 5, 10, and 20 phases equally.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “To solve this problem, many work [4, 5,11,47,77] propose to review knowledge by rehearsal-based mechanism.”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “In this paper, we consider the data-free incremental learning [36], where the training samples from previous”
A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning (2022, background): “[46, 8] release the burden of exemplars by data-free knowledge distillation [33].”
Deep Class-Incremental Learning: A Survey (2023, background): “1 Knowledge Distillation Logit Distillation [32], [85], [86], [87], [88], [89], [90], [91], [92]”
Memory-Free Generative Replay For Class-Incremental Learning (2021, background): “To tackle this issue, [29] and [6] employ synthetic images that generates by data-free knowledge distillation methods [5, 32] to alleviate catastrophic forgetting.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “Existing CL methods [67], [68] fall into three main categories: rehearsal-based [69], [70], parameter-based regularization [71], [72], and generative-based [67], [68], [73].”
TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation (2023, background): “not permitted to store data from previous tasks due to privacy and policy concerns and data will not be kept for a long time [21, 53, 34, 50].”
Class-Incremental Learning: A Survey (2023, background): “…[33], [73], [74], [75], [76], [77], [78], [79], [80] § 3.5 Knowledge Distillation Logit Distillation [81], [82], [83], [84], [85], [86], [87], [88] Feature Distillation [89], [90], [91], [92], [93], [94], [95], [96], [97], [98] Relational Distillation [27], [99], [100], [101], [102],…”
Biological underpinnings for lifelong learning machines (2022, background): “These can be broadly organized into three types: ‘rehearsal’, which store or generate data from past tasks for replay 11–13 ; ‘architectural’, which expand the model parameters 14–17 ; and ‘regularization-based’ approaches, which penalize changes to parameters important to past tasks 18–20 or use…”
Divide and not forget: Ensemble of selectively trained experts in Continual Learning (2024, methodology): “Then, the hardest scenario exemplar-free CIL is considered, where number of methods exists: LwF (Li & Hoiem, 2016), SDC (Yu et al., 2020), ABD (Smith et al., 2021), PASS (Zhu et al., 2021b), IL2A (Zhu et al., 2021a), SSRE (Zhu et al., 2022), FeTrIL (Petit et al., 2023).”