Generative Adversarial Network Training is a Continual Learning Problem

By Kevin J Liang, Chunyuan Li, Guoyin Wang, Lawrence Carin. 2018

53 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have proven to be a powerful framework for learning to draw samples from complex distributions. However, GANs are also notoriously difficult to train, with mode collapse and oscillations a common problem. We hypothesize that this is at least in part due to the evolution of the generator distribution and the catastrophic forgetting tendency of neural networks, which leads to the discriminator losing the ability to remember synthesized samples from previous instantiations of the generator. Recognizing this, our contributions are twofold. First, we show that GAN training makes for a more interesting and realistic benchmark for continual learning methods evaluation than some of the more canonical datasets. Second, we propose leveraging continual learning techniques to augment the discriminator, preserving its ability to recognize previous generator samples. We show that the resulting methods add only a light amount of computation, involve minimal changes to the model, and result in better overall performance on the examined image and text generation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “[15] counter discriminator forgetting by leveraging techniques from continual learning directly (Elastic Weight Sharing [11] and Intelligent Synapses [37]).”
Catastrophic forgetting and mode collapse in GANs (2020, background): “[15] independently came up with a similar intuition that GAN training is a continual learning problem.”
Task Agnostic Continual Learning via Meta Learning (2019, methodology): “Indeed, it has been shown empirically by Thanh-Tung & Tran (2019); Liang et al. (2019) that GANs suffer from catastrophic forgetting, and they adapted CL methods such as Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) and Synaptic Intelligence (SI) (Zenke et al., 2017) to alleviate…”
Efficient Feature Transformations for Discriminative and Generative Continual Learning (2021, background): “As in classification, if a generative adversarial network (GAN) [9] is trained on a sequence of different data distributions, it will experience catastrophic forgetting [26, 63, 4, 50], tending to forget how to produce samples from previous distributions.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “[182] think that GANs training is a continual learning problem [183].”
Feature Quantization Improves GAN Training (2020, background): “(iii) In particular, the distribution of fake samples shifts as the generator changes during training, making the classification task for discriminator evolve over time (Chen et al., 2019; Liang et al., 2018).”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, background): “CF has been also studied in the context of GANs (Liang et al., 2018; Li et al., 2020; Thanh-Tung & Tran, 2020).”
Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors (2021, methodology): “EWC (Kirkpatrick et al., 2017) reﬁnes this by weighting the L 2 by parameter importance, using the Fisher Information; Online EWC (Schwarz et al., 2018; Liang et al., 2018b) uses an online version that provides better scaling.”
Bayesian Nonparametric Weight Factorization for Continual Learning (2020, methodology): “EWC [19] reﬁnes this by weighting the L 2 by parameter importance, using the Fisher Information; Schwarz et al. [40], Liang et al. [27] propose an online version that provides better scaling ( Online EWC ).”