Quantitatively Evaluating GANs With Divergences Proposed for Training

By Daniel Jiwoong Im, He Ma, Graham W. Taylor, Kristin Branson. 2018

73 citations recorded by the source corpus.

Generative adversarial networks (GANs) have been extremely effective in approximating complex distributions of high-dimensional, input data samples, and substantial progress has been made in understanding and improving GAN performance in terms of both theory and application. However, we currently lack quantitative methods for model assessment. Because of this, while many GAN variants are being proposed, we have relatively little understanding of their relative abilities. In this paper, we evaluate the performance of various types of GANs using divergence and distance functions typically used only for training. We observe consistency across the various proposed metrics and, interestingly, the test-time metrics do not favour networks that use the same training-time criterion. We also compare the proposed metrics to human perceptual scores.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, methodology): “While the FID metric was introduced in response to Inception Score’s criticisms, and was shown empirically to alleviate some of the concerns with it (Im et al. 2018; Xu et al. 2018a; Lucic et al. 2018), it still suffers from problems in the layout-toimage setting.”
Assessing Generative Models via Precision and Recall (2018, methodology): “Another approach is to train a classifier between the real and fake distributions and to use its accuracy on a test set as a proxy for the quality of the samples [11, 17].”
Continual Learning for Robotics (2019, methodology): “More methods for evaluating generative models are described and assessed more in depth in [11, 61]; however, they have never been used in a CL setting.”
Pros and Cons of GAN Evaluation Measures (2018, background): “[43] proposed to evaluate GANs by exploring the divergence and distance metrics that were used during training GANs.”
Transferring GANs: generating images from limited data (2018, methodology): “In our experiments we use two recent metrics that show better correlation in recent studies [21, 4].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “[161] quantitatively evaluated GANs with divergences proposed for training.”
GILBO: One Metric to Measure Them All (2018, background): “endent of data, the GILBO does not directly measure sample quality, but extreme values (either low or high) correlate with poor sample quality, as demonstrated in the experiments below. Similarly, in Im et al. (2018), the authors propose using various GAN training objectives to quantitatively measur”
On GANs and GMMs (2018, background): “) Another criticism is the lack of a robust and consistent evaluation method for GANs [18, 10, 28].”
Synthesis of Realistic ECG using Generative Adversarial Networks (2019, methodology): “This includes two-sample tests such as maximum mean discrepancy, alternative loss functions such as the Wasserstein distance and inception scores [32, 33, 34].”
Multi-objective de novo drug design with conditional graph generative model (2018, methodology): “ergence(D JS): D JS(p gjjp data) = 1 2 D KL(p gjj p g+ p data 2 )+ 1 2 D KL(p datajj p g+ p data 2 ) (19) D KLand D JSare widely used in deep generated models for both training [17,48] and evaluation [49]. Here, the two values are determined using kernel density method implemented in SciPy [50]. We ”
Revisiting Precision and Recall Definition for Generative Model Evaluation (2019, methodology): “Finally, in (Im et al., 2018), classiﬁers trained with various divergences (normally used as objectives for discriminators during GAN training) are used to deﬁne a metric between Q and P .”
An Improved Evaluation Framework for Generative Adversarial Networks (2018, methodology): “Although FID was proved to be robust to mode dropping and mode collapse in recent literature [9,12,13,14], we argue that experimental demonstrations on its robustness in previous work”
Fair Generative Modeling via Weak Supervision (2019, background): “, 2016; Mohamed & Lakshminarayanan, 2016; Grover & Ermon, 2018), evaluation metrics based on two-sample tests (Gretton et al., 2007; Bowman et al., 2015; Lopez-Paz & Oquab, 2016; Danihelka et al., 2017; Rosca et al., 2017; Im et al., 2018; Gulrajani et al., 2018), or improved Monte Carlo inference v”
Detecting Overfitting of Deep Generative Networks via Latent Recovery (2019, methodology): “That is, they train a neural net to differentiate generated samples from real samples and similar to [18], use the resulting divergence as a measure of quality.”
Training Discriminative Models to Evaluate Generative Ones (2018, methodology): “Diﬀerent quantitative evaluation have also been experimented in [9] which compares models of GANs in various settings.”