GAN Memory with No Forgetting

By Yulai Cong, Miaoyun Zhao, Jianqiao Li, Sijia Wang, Lawrence Carin. 2020

151 citations recorded by the source corpus.

Seeking to address the fundamental issue of memory in lifelong learning, we propose a GAN memory that is capable of realistically remembering a stream of generative processes with \emph{no} forgetting. Our GAN memory is based on recognizing that one can modulate the ``style'' of a GAN model to form perceptually-distant targeted generation. Accordingly, we propose to do sequential style modulations atop a well-behaved base GAN model, to form sequential targeted generative models, while simultaneously benefiting from the transferred base knowledge. Experiments demonstrate the superiority of our method over existing approaches and its effectiveness in alleviating catastrophic forgetting for lifelong classification problems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “GAN-Memory [167] takes advantage of FiLM [201] and AdaFM [202] to learn task-speciﬁc parameters for each layer of a pre-trained generative model, while Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, background): “The first line of research [10,25,24] requires a generator to be trained on the original”
Constrained Few-shot Class-incremental Learning (2022, background): “Alternatively, the exemplars of past classes can be generated using GANs with extra computational cost [8].”
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (2021, background): “…from the previous tasks in an episodic memory (Rebufﬁ et al., 2017; Castro et al., 2018; Chaudhry et al., 2019; Wu et al., 2019; Kim et al., 2019) or keep a generative model that is trained to generate previous task samples (Shin et al., 2017; Wu et al., 2018; Hu et al., 2019; Cong et al., 2020).”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “We kindly acknowledge that recent works which replay from a generator (close to our setting) also use small variants of ImageNet in their experiments [1, 3, 8].”
Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “While we enforce this stringent condition in this paper, our approach may be easily combined with memory-based techniques such as coresets or generative replay [8, 10, 13, 31, 33, 35, 37, 41, 42, 44, 47].”
Rainbow Memory: Continual Learning with a Memory of Diverse Samples (2021, methodology): “Also, Borsos et al . propose a coreset generation method for the representative memory using cardinality-constrained bi-level optimization [3]. and Cong et al . propose a GAN based memory which they can perturb styles of remembered samples for incremental learning [8].”
Distilling Causal Effect of Data in Class-Incremental Learning (2021, background): “In fact, even though that the ever-increasing extra storage in data replay contradicts with the requirement of CIL, it is still the most reliable solution for antiforgetting [35, 45, 5].”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, background): “in slightly better performance than that of iCaRL[56]. However, the performance signiﬁcantly drops when relying exclusively on artiﬁcially generated images. Alternately, GAN Memory with No Forgetting [15] is based on sequential style modulations to represent the past memory by forming a sequential t”
Efficient Feature Transformations for Discriminative and Generative Continual Learning (2021, background): “As in classification, if a generative adversarial network (GAN) [9] is trained on a sequence of different data distributions, it will experience catastrophic forgetting [26, 63, 4, 50], tending to forget how to produce samples from previous distributions.”
Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners (2022, methodology): “To overcome catastrophic forgetting in artificial neural networks, three types of methods have been explored in the past few years, replay [15, 3, 6, 20, 10], regularization [9, 19, 8, 4, 17, 13, 5, 7, 16], and expand [22, 14, 1, 21].”
LFS-GAN: Lifelong Few-Shot Image Generation (2023, methodology): “The above example assumes that the rank r =2.”
Memory-Free Generative Replay For Class-Incremental Learning (2021, background): “Recently, GAN memory [7] proposes a sequential style modulation to transfer base knowledge to new tasks simultaneously with a well-behaved GAN model to achieve no forgetting even with a long task-sequence.”
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 (2022, background): “Most of the prior work on generative replay has focused on the problem of class-incremental classification van de Ven & Tolias (2019); Shin et al. (2017), or class-incremental generative modelling Cong et al. (2020) as a subroutine thereof, but significantly less exploration of GR in LRL.”
Class-Incremental Learning with Generative Classifiers (2021, background): “Some recent studies have shown competitive performance with generative replay on class-incremental learning problems with natural images [11, 30, 50], but the approaches in those studies depend on pre-trained networks (or on an extensive, nonincremental initialization phase [30]).”
SDDGR: Stable Diffusion-Based Deep Generative Replay for Class Incremental Object Detection (2024, methodology): “Among these, Replay methods are most frequently utilized for their simple yet powerful effects and can be categorized into two types: partial experience replay (ER) [2, 4, 14, 27, 44, 47] and generative replay (GR) [6, 12, 23, 58, 66, 67].”