Assessing Generative Models via Precision and Recall

By Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, Sylvain Gelly. 2018

758 citations recorded by the source corpus.

Recent advances in generative modeling have led to an increased interest in the study of statistical divergences as means of model comparison. Commonly used evaluation methods, such as the Frechet Inception Distance (FID), correlate well with the perceived quality of samples and are sensitive to mode dropping. However, these metrics are unable to distinguish between different failure cases since they only yield one-dimensional scores. We propose a novel definition of precision and recall for distributions which disentangles the divergence into two separate dimensions. The proposed notion is intuitive, retains desirable properties, and naturally leads to an efficient algorithm that can be used to evaluate generative models. We relate this notion to total variation as well as to recent evaluation metrics such as Inception Score and FID. To demonstrate the practical utility of the proposed approach we perform an empirical study on several variants of Generative Adversarial Networks and Variational Autoencoders. In an extensive set of experiments we show that the proposed metric is able to disentangle the quality of generated samples from the coverage of the target distribution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, background): “Instead, similarly to Sajjadi et al. (2018), we discretize the representations by clustering them into a large number of clusters.”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Generative Evaluation Evaluation of generative quality is a difﬁcult goal with numerous proxy metrics, such as reconstruction error, FID (Heusel et al., 2017), IS (Sali-mans et al., 2016), precision-recall (Sajjadi et al., 2018), log likelihood (for auto-regressive models), or the ELBO (for VAEs).”
Are GANs Created Equal? A Large-Scale Study (2017, background): “As such, developing quantitative evaluation metrics is a critical research direction [3, 23].”
Evaluating Lossy Compression Rates of Deep Generative Models (2020, background): “In order to address this, Sajjadi et al. (2018) proposed to study the precision-recall tradeoff for evaluating generative models.”
Improved Techniques for Training Score-Based Generative Models (2020, background): “However, we note that Inception and FID scores have known issues [21, 22] and they should be interpreted with caution as they may not correlate with visual quality in the expected way.”
Training Generative Adversarial Networks with Limited Data (2020, background): “, [5]) that FID roughly corresponds to Recall and IS to Precision, two independent aspects of result quality [36, 26].”
Analyzing and Improving the Image Quality of StyleGAN (2019, background): “Precision and Recall (P&R) [31, 22] provide additional visibility by explicitly quantifying the percentage of generated images that are similar to training data and the percentage of training data that can be generated, respectively.”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, background): “Improving sample quality by rejection sampling has been previously explored for GANs [1] as well as for VAEs [3] which combines a learned rejecting sampling proposal with the prior in order to reduce its gap with the aggregate posterior.”
Projected GANs Converge Faster (2021, methodology): “In the appendix, we also report other metrics that are less benchmarked in GAN literature: KID [3], SwAV-FID [39], precision and recall [51].”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “Commonly-used ones include log-likelihood [28, 41], density estimate with Parzen window [28], Inception Score [64], Perceptual Path Length [37], Fréchet Inception Distance (FID) [30], Classification Accuracy Score and its early variants [60, 64], Classifier Twosample Tests [44, 47], precision and re”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “Great research interest has been drawn on the evaluation of GANs (DeVries et al., 2019; Gurumurthy et al., 2017; Sajjadi et al., 2018; Snell et al., 2017; Xiang & Li, 2017).”
Improved Precision and Recall Metric for Assessing Generative Models (2019, methodology): “’s method [23] as well as the widely used FID metric [10].”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, background): “FID is shown to be sensitive to both the addition of spurious modes and to mode dropping [29, 30].”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, background): “Sajjadi et al. (2018) also examines diversity and quality (which they call precision and recall) in the context of generative image models.”
Classification Accuracy Score for Conditional Generative Models (2019, background): “Even more recent work [26, 27] calculate 1 and 2 independently by calculating precision and recall.”
Counterfactuals uncover the modular structure of deep generative models (2018, background): “Learning the generator parameters such that Y M precisely matches the support of the target data distribution is arguably a major goal for generative models (see e.g. Sajjadi et al. [2018]).”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, background): “These works instead focus on direct metrics [7], [8], [41], [54]–[56] for assessing the diversity and quality of generated images.”
Lifelong Generative Modeling (2017, background): “While the Frechet distance, negative ELBO and IS log-likelihood estimate provide a glimpse into model performance, there exists no conclusive metric that captures the quality of unsupervised generative models [124, 108] and active research suggests a direct trade-off between perceptual quality and m”