How good is my GAN?

By Konstantin Shmelkov, Cordelia Schmid, Alahari Karteek. 2018

387 citations recorded by the source corpus.

Generative adversarial networks (GANs) are one of the most popular methods for generating images today. While impressive results have been validated by visual inspection, a number of quantitative criteria have emerged only recently. We argue here that the existing ones are insufficient and need to be in adequation with the task at hand. In this paper we introduce two measures based on image classification—GAN-train and GAN-test, which approximate the recall (diversity) and precision (quality of the image) of GANs respectively. We evaluate a number of recent GAN approaches based on these two measures and demonstrate a clear difference in performance. Furthermore, we observe that the increasing difficulty of the dataset, from CIFAR10 over CIFAR100 to ImageNet, shows an inverse correlation with the quality of the GANs, as clearly evident from our measures.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Pros and Cons of GAN Evaluation Measures (2018, methodology): “[93] proposed to compare class-conditional GANs with GAN-train and GAN-test scores using a neural net classifier.”
The Knowledge Within: Methods for Data-Free Model Compression (2019, methodology): “her equivalent or slightly lower ﬁnal accuracy compared to real data. We associate this gap with the synthetic data quality as well as failure to fully capture the dataset variety (i.e. mode-dropping [26]). 4.3 Analysis of Data Generation Schemes For this section we consider ResNet44 trained on Cifa”
You Only Need Adversarial Supervision for Semantic Image Synthesis (2020, background): “Recently, a series of metrics have been proposed to address this issue by measuring scores related to the concepts of precision and recall (Ravuri & Vinyals, 2019; Shmelkov et al., 2018; Sajjadi et al., 2018; Kynkäänniemi et al., 2019).”
Medical image data augmentation: techniques, comparisons and interpretations (2023, background): “In other words, it shows how uniform or wide the feature distribution of the synthetic image is (Shmelkov et al. 2018).”
Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019, methodology): “Classification Accuracy Score [52] introduce the Classification Accuracy Score (CAS) to evaluate conditional image generation models, similar to [53].”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, background): “Following previous works ( e.g. Lucic et al. (2017); Shmelkov et al. (2018)), new studies have investigated formulating good criteria for GAN evaluation, or have conducted systematic GAN benchmarks.”
Classification Accuracy Score for Conditional Generative Models (2019, methodology): “Our work most closely mirrors [16], but differs in a some key respects.”
cGANs with Multi-Hinge Loss (2019, neutral): “55 SNGAN [39] 9.”
LARGE: Latent-Based Regression through GAN Semantics (2021, methodology): “When applied to the ImageNet classification task, some works [40,47] demonstrated that performance sees only modest improvement when enriching the real set, and becomes poor when replacing it.”