On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation

By Gaurav Parmar, Richard Zhang, Jun-Yan Zhu. 2021

84 citations recorded by the source corpus.

Metrics for evaluating generative models aim to measure the discrepancy between real and generated images. The often-used Frechet Inception Distance (FID) metric, for example, extracts"high-level"features using a deep network from the two sets. However, we find that the differences in"low-level"preprocessing, specifically image resizing and compression, can induce large variations and have unforeseen consequences. For instance, when resizing an image, e.g., with a bilinear or bicubic kernel, signal processing principles mandate adjusting prefilter width depending on the downsampling factor, to antialias to the appropriate bandwidth. However, commonly-used implementations use a fixed-width prefilter, resulting in aliasing artifacts. Such aliasing leads to corruptions in the feature extraction downstream. Next, lossy compression, such as JPEG, is commonly used to reduce the file size of an image. Although designed to minimally degrade the perceptual quality of an image, the operation also produces variations downstream. Furthermore, we show that if compression is used on real training images, FID can actually improve if the generated images are also subsequently compressed. This paper shows that choices in low-level image processing have been an underappreciated aspect of generative modeling. We identify and characterize variations in generative modeling development pipelines, provide recommendations based on signal processing principles, and release a reference implementation to facilitate future comparisons.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Evaluation & Benchmarks

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “However, since different data processing pipelines might lead to different results [64], we also evaluate our models with the script provided by Dhariwal and Nichol [15].”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “This is for two reasons: first, some papers [21, 22, 19] compare against arbitrary subsets of the training set which are not readily available; and second, subtle implementation differences can affect the resulting FID values [38].”
Alias-Free Generative Adversarial Networks (2021, background): “Aliasing, despite being a subtle and critical issue [43], has received little attention in the GAN literature.”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, background): “Clean FID Parmar et al. (2021) examine the sensitivity of FID score (and also other scores such as KID) to inconsistent and often incorrect implementations across diﬀerent image processing libraries when training and evaluating generative models.”
Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation (2021, methodology): “We note that FID computed by (Parmar et al., 2022) reports a higher FID score compared to the original FID calculation 4 .”
Score Matching Model for Unbounded Data Score (2021, methodology): “On high-dimensional datasets, we compute FID scores based on [47].”
U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, methodology): “We then compare the generated images qualitatively and quantitatively using various metrics such as Fréchet Inception Distance (FID) [65] and Inception Score (IS) [75].”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, methodology): “Consequently, implementations of the Frechet Inception Distance (FID) [63], which is used to evaluate generative models, would report different scores for the same models [112].”
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 (2021, unknown): “We also want to make a note on how we compute FID for vidoe generators.”
Latte: Latent Diffusion Transformer for Video Generation (2024, unknown): “Furthermore, we select 2,048 sampled videos for calculating FVD and FID scores.”
Perception Prioritized Training of Diffusion Models (2022, methodology): “However, since both metrics are sensitive to the preprocessing [32], we use a correctly implemented library [32].”