Are GANs Created Equal? A Large-Scale Study

By Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, Olivier Bousquet. 2017

1,114 citations recorded by the source corpus.

Generative adversarial networks (GAN) are a powerful subclass of generative models. Despite a very rich research activity leading to numerous interesting GAN algorithms, it is still very hard to assess which algorithm(s) perform better than others. We conduct a neutral, multi-faceted large-scale empirical study on state-of-the art models and evaluation measures. We find that most models can reach similar scores with enough hyperparameter optimization and random restarts. This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes. To overcome some limitations of the current metrics, we also propose several data sets on which precision and recall can be computed. Our experimental results suggest that future GAN research should be based on more systematic and objective evaluation procedures. Finally, we did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN introduced in \cite{goodfellow2014generative}.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Fine-tuning opens up a much larger design and hyperparameter space, which makes it difficult to fairly evaluate and computationally expensive to compare a diverse set of techniques as discussed in other large scale empirical studies (Lucic et al., 2018; Choi et al., 2019).”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, positive): “In the quest to draw sound and robust conclusions, we join other recent papers in highlighting the under-acknowledged existence of conclusion replication failure [47, 36, 39, 26, 19].”
Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “The WGAN discriminator is motivated in terms of defining a critic function whose gradient with respect to its input is better behaved (smoother) than original GANs, making the optimization of the generator easier (Lucic et al., 2018).”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “We do not modify the discriminator or the loss function in any way, and our work is thus orthogonal to the ongoing discussion about GAN loss functions, regularization, and hyper-parameters [23, 41, 4, 37, 40, 33].”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “As a result, it has been used in recent GANs papers [9,37,77] as well as large-scale evaluation study [48], despite facing criticism about the fact that FID is a biased estimator and sensitive to the number of samples used in the evaluation [5, 14].”
Object-Centric Image Generation from Layouts (2020, methodology): “While the FID metric was introduced in response to Inception Score’s criticisms, and was shown empirically to alleviate some of the concerns with it (Im et al. 2018; Xu et al. 2018a; Lucic et al. 2018), it still suffers from problems in the layout-toimage setting.”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, background): “While there are many variations of the GAN objective function and its network architectures [23, 30], in this paper we focus on improving the discriminator network.”
Diagnosing and Enhancing VAE Models (2019, methodology): “To further probe these issues, we again adopt the neutral testing framework from (Lucic et al., 2018), and retrain each model as κ is varied.”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, neutral): “Therefore our re-ported FID values are lower than other generative models reported in [31].”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, background): “Robustness across hyperparameters GANs are fragile; changes to the hyperparameter settings have a substantial impact to their performance [30, 32].”
Generative adversarial networks (2014, neutral): “Very many different specific formulations of these costs are possible and so far most popular formulations seem to perform roughly the same.(18) In the original version of GANs, J(D) was defined to be the negative log-likelihood that the discriminator assigns to the real-vs-fake labels given the inp”
On Network Design Spaces for Visual Recognition (2019, background): “There has been an encouraging recent trend toward better reproducibility in machine learning [25, 22, 9].”
Assessing Generative Models via Precision and Recall (2018, methodology): “A notion of precision and recall has previously been introduced in [18] where the authors compute the distance to the manifold of the true data and use it as a proxy for precision and recall on a synthetic data set.”
Show Your Work: Improved Reporting of Experimental Results (2019, methodology): “Moreover, recent investigations into “state-ofthe-art” claims have found competing methods to only be comparable, without clear superiority, even against baselines (Reimers and Gurevych, 2017; Lucic et al., 2018; Li and Talwalkar, 2019); this has exposed the need for reporting more than a single poi”
AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, background): “However, the backbone architecture design of GANs has received relatively less attention, and was often considered a less signiﬁcant factor accounting for GAN performance [39, 32].”
GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks (2018, background): “Whether the choice of GAN architecture will aﬀect the quality of the augmentation is unclear, however there is evidence [14] to suggest that diﬀerent GAN architectures produce results which are, on average, not signiﬁcantly diﬀerent from each other.”
High-Fidelity Generative Image Compression (2020, background): “Training Stability and Losses Training GANs for unconditional or class-conditional image generation is notoriously hard [29, 36, 10].”