Memory Replay GANs: learning to generate images from new categories without forgetting

By Chenshen Wu, Luis Herranz, Xialei Liu, Yaxing Wang, Joost van de Weijer, B. Raducanu. 2018

203 citations recorded by the source corpus.

Previous works on sequential learning address the problem of forgetting in discriminative models. In this paper we consider the case of generative models. In particular, we investigate generative adversarial networks (GANs) in the task of learning new categories in a sequential fashion. We first show that sequential fine tuning renders the network unable to properly generate images from previous categories (i.e. forgetting). Addressing this problem, we propose Memory Replay GANs (MeRGANs), a conditional GAN framework that integrates a memory replay generator. We study two methods to prevent forgetting by leveraging these replays, namely joint training with replay and replay alignment. Qualitative and quantitative experimental results in MNIST, SVHN and LSUN datasets show that our memory replay approach can generate competitive images while significantly mitigating the forgetting of previous categories.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Meta & Continual Learning

Citations in the corpus

DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “The fourth method is generative replay [Shin et al., 2017, Lesort et al., 2018, Wu et al., 2018, Lesort et al., 2018], where a generative model is used as a memory to produce samples from previous tasks.”
Learnability and Algorithm for Continual Learning (2023, methodology): “Pseudoreplay methods generate pseudo replay data for old tasks to serve as the replay data (Kamra et al., 2017; Shin et al., 2017; Wu et al., 2018; Seff et al., 2017; Kemker & Kanan, 2018; Hu et al., 2019; Rostami et al., 2019; Ostapenko et al., 2019).”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, background): “Existing works in the literature can be grouped into three main categories: rehearsal-based methods [16], [30], [31], [32], [33], parameter isolation approaches [34], [35] and distillation-based techniques [14], [36], [37].”
Continual Learning via Local Module Composition (2021, methodology): “Continual learning methods typically address the problem of forgetting through parameter regularization [46, 70, 99], replay [89, 78, 3, 73, 54, 12, 97, 39, 81, 13] or dynamic architectures (and MoEs) [83, 87, 52, 85, 53, 41].”
Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference (2021, background): “…from the previous tasks in an episodic memory (Rebufﬁ et al., 2017; Castro et al., 2018; Chaudhry et al., 2019; Wu et al., 2019; Kim et al., 2019) or keep a generative model that is trained to generate previous task samples (Shin et al., 2017; Wu et al., 2018; Hu et al., 2019; Cong et al., 2020).”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[67] applied continual learning in class-conditional image generation.”
Dynamic VAEs with Generative Replay for Continual Zero-shot Learning (2021, methodology): “Reference[41] is an improved version of[33], where they used class-conditional GAN to generate data.”
Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “EWC [19] has been adapted from classification tasks to generative tasks of label-conditioned image generation [30,37], but they present limited capability in both remembering previous tasks and generating realistic images.”
On Learning the Geodesic Path for Incremental Learning (2021, background): “A memory replay can also be achieved with a generative mechanism as proposed in [19, 35, 41], but this approach needs additional networks which are not easy to train and optimize.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…learning, while the dynamic generative memory model (Ostapenko et al., 2019), the deep generative replay model (Shin et al., 2017), the Memory Replay GAN model (Wu et al., 2018), and the Closed-Loop GAN model (Rios & Itti, 2019) are all used to continually learn to generate images and scenes.”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, methodology): “When applying parameters update the former freezes the previous network while the latter retrain part of it; (iii) Rehearsal based aims at preventing catastrophic forgetting by storing and replaying information from previous episodes (Rebuffi et al., 2017; Castro et al., 2018; Wu et al., 2018; Hou e”
ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning (2021, methodology): “Existing generative replay methods [35, 43, 28, 45, 18] often offline sample generated data or embedding, or save the old generators, which results in additional time and storage cost.”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, neutral): “*Currently working at Google memory replay [41, 1].”
Online Continual Learning under Extreme Memory Constraints (2020, methodology): “opose ICARL where a strategy to select exemplars in combination with a distillation loss is introduced. Subsequent works [7,31] further analyze exemplars selection strategies. Dierently, other works [24,29,30] propose to employ generative networks to generate synthetic data of old tasks. However, th”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, methodology): “hes are using generative models, such as either Generative Adversarial Networks (GANs) [21] or Variational Autoencoders (VAEs) [32]) to replay the previously learnt knowledge. For instance, Wu et al. [64] proposed a novel lifelong generative framework, namely Memory Replay GANs (MeRGANs), which main”
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning (2020, methodology): “Approaches include methods that mitigate catastrophic forgetting via replay of examplars (iCarl (Rebuffi et al., 2017), EEIL (Castro et al., 2018), and GEM (Lopez-Paz & Ranzato, 2017)) or by performing pseudo-replay with GAN-generated data (Shin et al., 2017; Wu et al., 2018).”
Generative Feature Replay For Class-Incremental Learning (2020, methodology): “These samples could be real ones (’exemplars’), like in [4, 25, 41] in which we refer to the process as ’rehearsal’ or could be synthethic ones obtained through generative mechanisms, in which case we refer to the process as ’pseudo-rehearsal’ [43, 46, 49].”