Reliable Fidelity and Diversity Metrics for Generative Models

By Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, Jaejun Yoo. 2020

603 citations recorded by the source corpus.

Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Frechet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the generated images, recent papers have introduced variants of precision and recall metrics to diagnose those properties separately. In this paper, we show that even the latest version of the precision and recall metrics are not reliable yet. For example, they fail to detect the match between two identical distributions, they are not robust against outliers, and the evaluation hyperparameters are selected arbitrarily. We propose density and coverage metrics that solve the above issues. We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics. Code: this https URL.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “The efficacy of random features for evaluating generative models has been demonstrated in [Naeem et al. 2020].”
How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models (2021, methodology): “R OBUSTNESS TO OUTLIERS Naeem et al. (2020) showed that the precision and recall metrics as proposed by (Sajjadi et al., 2018; Kynk¨a¨anniemi et al., 2019) are not robust to outliers.”
The Role of ImageNet Classes in Fréchet Inception Distance (2022, methodology): “This effect is difﬁcult to quantify because the current widespread metrics (KID and Precision/Recall) also rely on the feature spaces of ImageNet classiﬁers.”
The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022, background): “Several other variations of precision and recall have been proposed (Kynkäänniemi et al., 2019; Simon et al., 2019; Naeem et al., 2020).”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, background): “[45] argue that even the latest version of the precision and recall metrics are still not reliable as they a) fail to detect the match between two identical distributions, and b) are not robust against outliers, and have arbitrarily selected evaluation hyperparameters.”
Few-shot Image Generation via Cross-domain Correspondence (2021, methodology): “To overcome this, density and coverage metrics were proposed to evaluate the generative models [7].”
Instance-Conditioned GAN (2021, positive): “Results are provided on the ImageNet dataset, following the experimental setup proposed in [41].”
Using latent space regression to analyze and leverage compositionality in GANs (2021, methodology): “Due to the smaller sample size, we use density here as a measure of realism (higher is better), which measures proximity to the real-image manifold (Naeem et al., 2020) and compare to L1 reconstruction (Eqn.”
Instance Selection for GANs (2020, methodology): “We use a variety of evaluation metrics to diagnose the eﬀect that training with instance selection has on the learned distribution, including: (1) Inception Score (IS) [22], (2) Fréchet Inception Distance (FID) [9], (3) Precision and Recall (P&R) [13], and (4) Density and Coverage (D&C) [18].”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “We use the Density and Coverage metrics proposed in Naeem et al. (2020), and compute them using the code provided by the authors.”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, neutral): “4) CLIP-R-Precision: CLIP-R-Precision is an assessment metric to address the model-speciﬁc character of the R-Precision metric [69].”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Besides the IS and FID, there have been multiple other proposals such the detection based score [140], SceneFID [103], the classification accuracy score (CAS) [139], precision and recall metrics [166, 167], and the recently proposed density and coverage metrics [168], which have not yet been adopted”