Memorization Precedes Generation: Learning Unsupervised GANs with Memory Networks

By Youngjin Kim, Minjung Kim, Gunhee Kim. 2018

45 citations recorded by the source corpus.

We propose an approach to address two issues that commonly occur during training of unsupervised GANs. First, since GANs use only a continuous latent distribution to embed multiple classes or clusters of data, they often do not correctly handle the structural discontinuity between disparate classes in a latent space. Second, discriminators of GANs easily forget about past generated samples by generators, incurring instability during adversarial training. We argue that these two infamous problems of unsupervised GAN training can be largely alleviated by a learnable memory network to which both generators and discriminators can access. Generators can effectively learn representation of training samples to understand underlying cluster distributions of data, which ease the structure discontinuity problem. At the same time, discriminators can better memorize clusters of previously generated samples, which mitigate the forgetting problem. We propose a novel end-to-end GAN model named memoryGAN, which involves a memory network that is unsupervisedly trainable and integrable to many existing GAN models. With evaluations on multiple datasets such as Fashion-MNIST, CelebA, CIFAR10, and Chairs, we show that our model is probabilistically interpretable, and generates realistic image samples of high visual fidelity. The memoryGAN also achieves the state-of-the-art inception scores over unsupervised GAN models on the CIFAR10 dataset, without any optimization tricks and weaker divergences.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Self-Supervised GANs via Auxiliary Rotation Loss (2018, background): “This issue is usually addressed either by reusing old samples or by applying continual learning techniques [14, 15, 16, 17, 18, 19].”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, methodology): “Previous approaches to mitigate catastrophic forgetting in GANs include using forms of memory [8], [57], [58], such as checkpoint averaging.”
Feature Quantization Improves GAN Training (2020, background): “Kim et al. (2018) increased the model complexity via proposing a shared and sophisticated memory module for both generator and discriminator.”
Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (2020, methodology): “o an auto-encoder to remember normal events for anomaly detection [13]. Kim et al. introduced the memory network into GANs to remember previously generated samples to alleviate the forgetting problem [22]. However, the memories in the above works are dened as attened vectors, thus the spatial struct”
Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection (2019, methodology): “The external memory has also been used for multi-modal data generation [15, 21], for circumventing the mode collapse issue and preserving detailed data structure.”