Seeing What a GAN Cannot Generate

By David Bau, Jun-Yan Zhu, Jonas Wulff, William S. Peebles, Hendrik Strobelt, Bolei Zhou, Antonio Torralba. 2019

353 citations recorded by the source corpus.

Despite the success of Generative Adversarial Networks (GANs), mode collapse remains a serious issue during GAN training. To date, little work has focused on understanding and quantifying which modes have been dropped by a model. In this work, we visualize mode collapse at both the distribution level and the instance level. First, we deploy a semantic segmentation network to compare the distribution of segmented objects in the generated images with the target distribution in the training set. Differences in statistics reveal object classes that are omitted by a GAN. Second, given the identified omitted object classes, we visualize the GAN's omissions directly. In particular, we compare specific differences between individual photos and their approximate inversions by a GAN. To this end, we relax the problem of inversion and solve the tractable problem of inverting a GAN layer instead of the entire generator. Finally, we use this framework to analyze several recent GANs trained on multiple datasets and identify their typical failure cases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, background): “[5] suggested that GANs have limited generation capacity, and analyzed the image structures that a pretrained GAN is unable to produce.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “However, these methods have some trouble manipulating real images because of imperfect reconstruction [107, 7].”
Denoising Diffusion Implicit Models (2020, background): “It is also relevant to investigate whether DDIMs exhibit other properties of existing implicit models (Bau et al., 2019).”
Towards causal generative scene models via competition of experts (2020, background): “…impressive on those tasks Karras et al. (2019a;b)—generating natural scenes containing several objects in non-trivial constellations gets increasingly difﬁcult within this framework due to the combinatorial number of compositions that need to be represented and reasoned about (Bau et al., 2019).”
Interpreting the Latent Space of GANs for Semantic Face Editing (2019, methodology): “To make GANs applicable for real image processing, existing methods proposed to reverse the mapping from the latent space to the image space [27, 39, 24, 5, 16] or learn an additional encoder associated with the GAN training [13, 12, 38].”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, background): “Such embedding algorithms can be used to analyze the limitations of GANs [5], do image inpainting [8, 39, 38, 36], local image editing [40, 17], global image transformations such as image morphing and expression transfer [1], and few-shot video generation [35, 34].”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, methodology): “Hybrid methods combine these ideas and initialize iterative optimization from an encoder prediction [99, 5, 3, 4, 98].”
When and How Can Deep Generative Models be Inverted? (2020, methodology): “An alternative approach is to train an encoding neural network that maps images to their latent vectors [28, 8, 2, 23]; however, this method is not accompanied by any theoretical justiﬁcation.”
Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models (2019, neutral): “The idea of using a different embedding space than the output of the Generator it is not new; activations from VGG16 [7] have also been used for inversion [1].”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, methodology): “They also require GAN inversion, which can be inefficient and inaccurate for various datasets (Huh et al., 2020; Karras et al., 2020b; Bau et al., 2019b; Xu et al., 2021).”
Semantic Pyramid for Image Generation (2020, methodology): “Differently from classical GANs, we perform image to image mapping using a conditional GAN similarly to [19, 36].”
Unsupervised 3D Shape Completion through GAN Inversion (2021, background): “This appealing nature of GAN prior has been extensively exploited on various image restoration and manipulation tasks [3, 2, 25, 13].”
EditGAN: High-Precision Semantic Image Editing (2021, methodology): “Previous papers studied encoder-based methods [58, 59, 60, 61, 62], used primarily optimization-based techniques [63, 64, 65, 66, 67, 68, 69, 26], and developed hybrid approaches [63, 24, 25, 70, 71].”
Learning Object Placement by Inpainting for Compositional Data Augmentation (2020, background): “After the training, the inpainting network learns a prior to ﬁll the holes with background information even if the holes were previously occupied by some objects, which has been studied in [2].”
A Simple Baseline for StyleGAN Inversion (2021, neutral): “Existing GAN inversion methods can be subdivided into optimization-based [1, 2, 15, 34], forwardbased [35, 33, 6, 16] and hybrid [43, 8, 7] approaches.”
Image Processing Using Multi-Code GAN Prior (2019, methodology): “The other is to train an extra encoder to learn the mapping from the image space to the latent space [33, 50, 6, 5].”
Diverse Image Generation via Self-Conditioned GANs (2020, methodology): “While random samples reveal the positive output of the generator, reconstructions of training set images [44, 6] can be used to visualize the omissions of the generator [3].”
What makes fake images detectable? Understanding properties that generalize (2020, methodology): “combine two datasets – random samples from the generator, as well as images obtained by reprojecting the real images into the GAN following [3].”