Continual Learning in Generative Adversarial Nets

By Ari Seff, Alex Beatson, Daniel Suo, Han Liu. 2017

135 citations recorded by the source corpus.

Developments in deep generative models have allowed for tractable learning of high-dimensional data distributions. While the employed learning procedures typically assume that training data is drawn i.i.d. from the distribution of interest, it may be desirable to model distinct distributions which are observed sequentially, such as when different classes are encountered over time. Although conditional variations of deep generative models permit multiple distributions to be modeled by a single network in a disentangled fashion, they are susceptible to catastrophic forgetting when the distributions are encountered sequentially. In this paper, we adapt recent work in reducing catastrophic forgetting to the task of training generative adversarial networks on a sequence of distinct distributions, enabling continual generative modeling.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Self-Supervised GANs via Auxiliary Rotation Loss (2018, background): “This issue is usually addressed either by reusing old samples or by applying continual learning techniques [14, 15, 16, 17, 18, 19].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Weight regularization [314], [374], [427], [428] and experience replay [157], [428] have been shown to be effective in mitigating catastrophic forgetting of generative models.”
Variational Continual Learning (2017, background): “The approach could be extended to generative adversarial networks (GAN) (Goodfellow et al., 2014b) for which continual learning is an open problem, but see Seﬀ et al. (2017) for an initial attempt.”
Learnability and Algorithm for Continual Learning (2023, methodology): “Pseudoreplay methods generate pseudo replay data for old tasks to serve as the replay data (Kamra et al., 2017; Shin et al., 2017; Wu et al., 2018; Seff et al., 2017; Kemker & Kanan, 2018; Hu et al., 2019; Rostami et al., 2019; Ostapenko et al., 2019).”
A Theoretical Study on Solving Continual Learning (2022, background): “Some systems learn to generate pseudo training data of old tasks to be used in replay, called pseudo-replay [41, 42, 43, 44, 45, 46, 21, 47, 48].”
Continual Training of Language Models for Few-Shot Learning (2022, methodology): “(1) Regularization methods (Kirkpatrick et al., 2016; Seff et al., 2017) add a regularization to ensure minimal changes to weights for previous tasks.”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, background): “Related work includes (Shin et al. 2017; Wu et al. 2018; Kamra, Gupta, and Liu 2017; Seff et al. 2017; Hu et al. 2019; Ostapenko et al. 2019; Hayes et al. 2019; von Oswald et al. 2020).”
Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, background): “…models focus on (1), i.e., dealing with catastrophic forgetting or simply forgetting (Chen and Liu, 2018; Parisi et al., 2019; Li and Hoiem, 2016; Seff et al., 2017; Shin et al., 2017; Kirkpatrick et al., 2016; Rebufﬁ et al., 2017; Yoon et al., 2018; He and Jaeger, 2018; Yoon et al., 2018; Masse…”
Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “(1) Regularization-based approaches, such as those in [27, 30, 51, 69], add a regularization in the loss to consolidate weights for previous tasks when learning a new task.”
Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks (2021, methodology): “Regularization-based methods, such as those in (Kirkpatrick et al., 2016; Lee et al., 2017; Seff et al., 2017), add a regularization in the loss to consolidate previous knowledge when learning a new task.”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “…learn to generate pseudo old data and use them to jointly train with the new task data (Shin et al. 2017; Wu et al. 2018; Kamra, Gupta, and Liu 2017; Seff et al. 2017; Hu et al. 2019; Lesort et al. 2018; Ostapenko et al. 2019; Hayes et al. 2019) (also known as rehearsal (von Oswald et al. 2020)).”
Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “EWC [19] has been adapted from classification tasks to generative tasks of label-conditioned image generation [30,37], but they present limited capability in both remembering previous tasks and generating realistic images.”
Rehearsal revealed: The limits and merits of revisiting samples in continual learning (2021, background): “Another way, called pseudo-rehearsal, learns the input distribution with a generative model [33, 34, 31].”
Rainbow Memory: Continual Learning with a Memory of Diverse Samples (2021, methodology): “Generative models are employed to generate past task samples [42, 41, 46, 21] instead of sampling.”
EEC: Learning to Encode and Regenerate Images for Continual Learning (2021, background): “Both EEC and EECS outperform EWC-M and DGR by significant margins on the MNIST on A5 and A10.”
Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams (2020, methodology): “The replay data is either obtained directly from operational memoryM with stored samples [37, 31, 48, 18] or generated using generative models [42, 23, 39, 50].”
Catastrophic forgetting and mode collapse in GANs (2020, background): “[14] studied the standard continual learning setting in which a GAN is trained to generate samples from a set of distributions introduced sequentially.”
GAN Memory with No Forgetting (2020, methodology): “Reviewing existing methods, [77] revealed generative replay (or pseudo-rehearsal) [72, 69, 86, 66, 88] is an effective and general strategy for lifelong learning, with this further supported by [40, 78].”