An empirical study on evaluation metrics of generative adversarial networks

By Qiantong Xu, Gao Huang, Yang Yuan, Chuan Guo, Yu Sun, Felix Wu, Kilian Q. Weinberger. 2018

306 citations recorded by the source corpus.

Evaluating generative adversarial networks (GANs) is inherently challenging. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. With a series of carefully designed experiments, we comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbor (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far they are from learning the target distribution.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, methodology): “While the FID metric was introduced in response to Inception Score’s criticisms, and was shown empirically to alleviate some of the concerns with it (Im et al. 2018; Xu et al. 2018a; Lucic et al. 2018), it still suffers from problems in the layout-toimage setting.”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, methodology): “l-wise independence, and can help to compare the visual appearance of different inpainting images. But in practice, they may assign favorable scores to perceptually inaccurate results. Recent studies [56] have shown that metrics based on deep features are closer to those based on human perception. T”
GAN Inversion: A Survey (2021, background): “In [82], Xu et al. present an empirical study on the evaluation metrics of GAN models.”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “It is able to distinguish generated images from real images, and both its sample complexity and computational complexity are low [39].”
Rethinking FID: Towards a Better Evaluation Metric for Image Generation (2023, methodology): “Generated image quality has been assessed using a variety of metrics including log-likelihood [9], Inception Score (IS) [1, 22], Kernel Inception Distance (KID) [2, 25], F ´ rechet Inception Distance (FID) [12], perceptual path length (PPL) [13], Gaussian Parzen window [9], and HYPE [27].”
An Improved Evaluation Framework for Generative Adversarial Networks (2018, methodology): “Although FID was proved to be robust to mode dropping and mode collapse in recent literature [9,12,13,14], we argue that experimental demonstrations on its robustness in previous work”
PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows (2019, background): “It has also been explored as a metric for evaluating GANs [50].”
A Non-Parametric Test to Detect Data-Copying in Generative Models (2020, methodology): “There has been a large body of prior work on the evaluation of generative models (Salimans et al., 2016; Lopez-Paz and Oquab, 2016; Richardson and Weiss, 2018; Sajjadi et al., 2018; Xu et al., 2018; Wu et al., 2017) .”
Synthesis of Realistic ECG using Generative Adversarial Networks (2019, methodology): “This includes two-sample tests such as maximum mean discrepancy, alternative loss functions such as the Wasserstein distance and inception scores [32, 33, 34].”
Synthetic-Neuroscore: Using a neuro-AI interface for evaluating generative adversarial networks (2019, background): “Traditional metrics need a large sample size to capture the underlying statistical properties of the real and generated images [12, 16].”
A Survey of Unsupervised Deep Domain Adaptation (2018, result): “[258] and Borji [19] survey and compare many of these GAN evaluation methods.”
Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories With GANs (2019, methodology): “For a more quantitative evaluation of generative models, we have used the following two metrics to assess the set of fake trajectories versus the set of real samples [21].”