Pros and Cons of GAN Evaluation Measures

By Ali Borji. 2018

987 citations recorded by the source corpus.

Generative models, in particular generative adverserial networks (GANs), have received a lot of attention recently. A number of GAN variants have been proposed and have been utilized in many applications. Despite large strides in terms of theoretical progress, evaluating and comparing GANs remains a daunting task. While several measures have been introduced, as of yet, there is no consensus as to which measure best captures strengths and limitations of models and should be used for fair model comparison. As in other areas of computer vision and machine learning, it is critical to settle on one or few good measures to steer the progress in this field. In this paper, I review and critically discuss more than 19 quantitative and 4 qualitative measures for evaluating generative models with a particular emphasis on GAN-derived models.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “Each metric has associated pros and cons [6, 72] and none are perfect.”
Assessing Generative Models via Precision and Recall (2018, background): “A recent review of popular approaches is presented in [5].”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “How to select an appropriate evaluation metric is still a difﬁcult problem, [43] presented several measures as meta-measures to guide researchers to choose quantitative evaluation metrics.”
Brain-inspired replay for continual learning with artificial neural networks (2020, background): “However, we note that these measures have several issues associated with them (1, 2) and that using them to compare between VAE models at the internal level versus at the pixel level might not be fair.”
Generative Models from the perspective of Continual Learning (2018, background): “For all the progress made in quantitative metrics for evaluating generative models [27], qualitative evaluation remains a widely used and informative method.”
Continual Learning for Robotics (2019, methodology): “More methods for evaluating generative models are described and assessed more in depth in (Borji, 2018; Jiwoong Im et al., 2018); however, they have never been used in a CL setting.”
The GAN Landscape: Losses, Architectures, Regularization, and Normalization (2018, background): “For an in-depth overview of quantitative metrics we refer the reader to (Borji, 2018).”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, methodology): “Even widely used metrics, such as Inception Score [52] and Fréchet Inception Distance [23], have been discredited for their application to non-ImageNet datasets [3, 48, 8, 46].”
The relativistic discriminator: a key element missing from standard GAN (2018, neutral): “We report the Fréchet Inception Distance (FID) [Heusel et al., 2017], a measure that is generally better correlated with data quality than the Inception Distance [Salimans et al., 2016] [Borji, 2018]; lower FID means that the generated images are of better quality.”
Transferring GANs: generating images from limited data (2018, methodology): “In our experiments we use two recent metrics that show better correlation in recent studies [21, 4].”
Evaluation of Mode Collapse in Generative Adversarial Networks (2018, background): “Several metrics have been proposed for evaluation of GANs for both synthetic and real datasets [10]–[12].”
PA-GAN: Improving GAN Training by Progressive Augmentation (2018, methodology): “From the comparison conducted in (Borji, 2018), FID is considered to be the most informative of the measures.”
MedGAN: Medical Image Translation using GANs (2018, methodology): “vely and quantitatively. With respect to quantitative experiments, there is no consensus in the scientiﬁc community regarding the best evaluation metrics to asses the performance of generative models [67]. Therefore, we utilize several traditional image quality metrics to judge the quality of the tr”
Generative Adversarial Networks: A Survey and Taxonomy (2019, background): “In this work. we are going to study existing GAN-variants that handle this aspect in the area of computer vision while those readers interested in the second aspect can consult [54], [62].”
Generating Multi-Categorical Samples with Generative Adversarial Networks (2018, background): “(Borji, 2018) provides a discussion of commonly used metrics.”
Detecting Overfitting of Deep Generative Networks via Latent Recovery (2019, background): “This progress has created a huge need to evaluate generated image quality, which to some degree has not been fully answered [6].”
An Improved Evaluation Framework for Generative Adversarial Networks (2018, background): “While Inception Score was the most popular metric in the last two years, it was believed to be misleading in recent literature [9,12,14,15,30].”