Effectively Unbiased FID and Inception Score and Where to Find Them

By Min Jin Chong, D.A. Forsyth. 2019

268 citations recorded by the source corpus.

This paper shows that two commonly used evaluation metrics for generative models, the Fréchet Inception Distance (FID) and the Inception Score (IS), are biased -- the expected value of the score computed for a finite sample set is not the true value of the score. Worse, the paper shows that the bias term depends on the particular model being evaluated, so model A may get a better score than model B simply because model A's bias term is smaller. This effect cannot be fixed by evaluating at a fixed number of samples. This means all comparisons using FID or IS as currently computed are unreliable. We then show how to extrapolate the score to obtain an effectively bias-free estimate of scores computed with an infinite number of samples, which we term FID Infinity and IS Infinity. In turn, this effectively bias-free estimate requires good estimates of scores with a finite number of samples. We show that using Quasi-Monte Carlo integration notably improves estimates of FID and IS for finite sample sets. Our extrapolated scores are simple, drop-in replacements for the finite sample scores. Additionally, we show that using low discrepancy sequence in GAN training offers small improvements in the resulting generator.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “As a result, it has been used in recent GANs papers [9,37,77] as well as large-scale evaluation study [48], despite facing criticism about the fact that FID is a biased estimator and sensitive to the number of samples used in the evaluation [5, 14].”
The Role of ImageNet Classes in Fréchet Inception Distance (2022, background): “…et al., 2018; Kynk ¨ a ¨ anniemi et al., 2019), and its numerical value is signiﬁcantly affected by various details, including the sample count (Binkowski et al., 2018; Chong & Forsyth, 2020), the exact instance of the feature network, and even low-level image processing (Parmar et al., 2022).”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, background): “Smaller sample sizes can lead to overestimating the actual FID [17].”
On Aliased Resizing and Surprising Subtleties in GAN Evaluation (2022, methodology): “As a result, it has been used in recent GANs papers [7,33,71] as well as large-scale evaluation study [44], despite facing criticism about the fact that FID is a biased estimator and sensitive to the number of samples used in the evaluation [4, 10].”
Lumiere: A Space-Time Diffusion Model for Video Generation (2024, unknown): “Chong & Forsyth (2020)), these metrics do not faithfully reflect human perception, and may be significantly influenced by low-level details (Parmar et al., 2022) and by the distribution shift between the reference UCF101 data and the T2V training data (Girdhar et al., 2023).”
A Continuous Time Framework for Discrete Denoising Models (2022, result): “Calculated with 10k samples, hence the discrepancy with Table 1 [27].”