Improved Precision and Recall Metric for Assessing Generative Models

By Tuomas Kynkaanniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, Timo Aila. 2019

1,312 citations recorded by the source corpus.

The ability to automatically estimate the quality and coverage of the samples produced by a generative model is a vital requirement for driving algorithm research. We present an evaluation metric that can separately and reliably measure both of these aspects in image generation tasks by forming explicit, non-parametric representations of the manifolds of real and generated data. We demonstrate the effectiveness of our metric in StyleGAN and BigGAN by providing several illustrative examples where existing metrics yield uninformative or contradictory results. Furthermore, we analyze multiple design variants of StyleGAN to better understand the relationships between the model architecture, training methods, and the properties of the resulting sample distribution. In the process, we identify new variants that improve the state-of-the-art. We also perform the first principled analysis of truncation methods and identify an improved method. Finally, we extend our metric to estimate the perceptual quality of individual samples, and use this to study latent space interpolations.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “We train unconditional models of 2562 images on CelebA-HQ [38], FFHQ [40], LSUN-Churches and -Bedrooms [98] and evaluate the i) sample quality and ii) their coverage of the data manifold using ii) FID [28] and ii) Precision-and-Recall [49].”
Training Generative Adversarial Networks with Limited Data (2020, background): “, [5]) that FID roughly corresponds to Recall and IS to Precision, two independent aspects of result quality [36, 26].”
Analyzing and Improving the Image Quality of StyleGAN (2019, methodology): “If we were willing to sacrifice scale-specific controls (see video), we could simply remove the normalization, thus removing the artifacts and also improving FID slightly [22].”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “Improving sample quality by rejection sampling has been previously explored for GANs [1] as well as for VAEs [3] which combines a learned rejecting sampling proposal with the prior in order to reduce its gap with the aggregate posterior.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, background): “We look at Precision/Recall (Kynk ¨ a ¨ anniemi et al., 2019), FID (Heusel et al., 2017), In-ception Score (Salimans et al., 2016), and CLIP score (Radford et al., 2021).”
Projected GANs Converge Faster (2021, methodology): “For our evalution, we use the improved formulation by [44].”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “GANs [16] currently hold the state-of-the-art on most image generation tasks [5, 60, 22] as measured by sample quality metrics such as FID [18], Inception Score [47] and Precision [26].”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “Commonly-used ones include log-likelihood [28, 41], density estimate with Parzen window [28], Inception Score [64], Perceptual Path Length [37], Fréchet Inception Distance (FID) [30], Classification Accuracy Score and its early variants [60, 64], Classifier Twosample Tests [44, 47], precision and re”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “Furthermore, by tuning the truncation ψ (Karras et al., 2019b; Kynkäänniemi et al., 2019) that explicitly amplifies the stochastic branch by ψ times, co-modulated GANs can easily trade-off between quality and diversity (see Fig.”
Scalable Diffusion Models with Transformers (2022, methodology): “For each of our DiT models trained for 400K iterations, we compute FID-10K using [16, 32, 64, 128, 256, 1000] sampling steps.”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “We consider Classiﬁcation Accuracy Score (CAS) [36] and Precision/Recall [30] as two metrics for evaluating sample diversity, in addition to sample quality.”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “To quantify mode coverage, we leverage the recall metric [Kynkäänniemi et al. 2019] and find that Config-B achieves a low recall of 0.”
Generating Images with Sparse Representations (2021, methodology): “We therefore use the sample-based Frechet inception distance (FID) (Heusel et al., 2017) and precision and recall (Kynkäänniemi et al., 2019) metrics, which enable comparison between any model that produces samples.”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, background): “These works instead focus on direct metrics [7,19,23,32,53,54] for assessing the diversity and quality of generated images.”
Instance Selection for GANs (2020, methodology): “An image manifold is more accurately deﬁned in regions where many data points are in close proximity to each other [13].”
Positional Encoding as Spatial Inductive Bias in GANs (2020, background): “2, by measuring Fréchet inception distantance (FID) [15] and the precision and recall [26], we investigate how zero padding influences StyleGAN2 [22] on FFHQ dataset [21].”
Self-Consuming Generative Models Go MAD (2023, methodology): “We compute recall in a manner similar to precision [39].”