On Aliased Resizing and Surprising Subtleties in GAN Evaluation

By Gaurav Parmar, Richard Zhang, Jun-Yan Zhu. 2022

546 citations recorded by the source corpus.

Metrics for evaluating generative models aim to measure the discrepancy between real and generated images. The often-used Fréchet Inception Distance (FID) metric, for example, extracts “high-level” features using a deep network from the two sets. However, we find that the differences in “low-level” preprocessing, specifically image resizing and compression, can induce large variations and have unforeseen consequences. For instance, when resizing an image, e.g., with a bilinear or bicubic kernel, signal processing principles mandate adjusting prefilter width depending on the downsampling factor, to antialias to the appropriate bandwidth. However, commonly-used implementations use a fixed-width prefilter, resulting in aliasing artifacts. Such aliasing leads to corruptions in the feature extraction down-stream. Next, lossy compression, such as JPEG, is commonly used to reduce the file size of an image. Although designed to minimally degrade the perceptual quality of an image, the operation also produces variations downstream. Furthermore, we show that if compression is used on real training images, FID can actually improve if the generated images are also subsequently compressed. This paper shows that choices in low-level image processing have been an under-appreciated aspect of generative modeling. We identify and characterize variations in generative modeling development pipelines, provide recommendations based on signal processing principles, and release a reference implementation to facilitate future comparisons.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

Scalable Diffusion Models with Transformers (2022, methodology): “FID is known to be sensitive to small implementation details [37]; to ensure accurate comparisons, all values re-ported in this paper are obtained by exporting samples and using ADM’s TensorFlow evaluation suite [9].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “Both FID and CLIP scores have limitations, for example FID is not fully aligned with perceptual quality [44], and CLIP is ineffective at counting [51].”
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, unknown): “We use the implementation of Clean-FID [66].”
Scaling up GANs for Text-to-Image Synthesis (2023, unknown): “We implement GigaGAN based on the StudioGAN PyTorch library [37], following the standard FID evaluation protocol with the anti-aliasing bicubic resize function [67], unless otherwise noted.”
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, unknown): “Importantly, as most embedding-based meth-ods tend to incorporate facial pose and expression into the representation, the generated images often lack variation in the facial region.”
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models (2022, background): “However, they do not necessarily align with actual user preference [26].”
MACE: Mass Concept Erasure in Diffusion Models (2024, methodology): “We sample 30,000 cap-tions from the validation set to generate images and evaluate FID [45] and CLIP score [47].”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, methodology): “The outputs are downsampled to 256 × 256 and compared with 40,504 real images from the same validation set using clean-FID [89].”
MagicVideo: Efficient Video Generation With Latent Diffusion Models (2022, methodology): “When comparing with other methods, we evaluate the zero-shot performance with text prompt from the test dataset of UCF101 [41], MSR-VTT [49] and calculate the Frechet Inception Distance (FID) [25] and Frechet Video Distance (FVD) [45] with reference to the images in their test dataset.”
The Role of ImageNet Classes in Fréchet Inception Distance (2022, background): “However, it conflates the resemblance to real data and the amount of variation to a single value [29, 47], and its numerical value is significantly affected by various details, including the sample count [3, 7], the architecture (and instance) of the feature network, and even low-level image process”
Make-A-Video: Text-to-Video Generation without Text-Video Data (2022, methodology): “Method Zero-Shot Samples/Input FID ( ↓ ) CLIPSIM ( ↑ ) GODIVA (Wu et al., 2021a) No 30 − 0 .”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “, FID is not always consistent with perceptual quality [16], [148], and CLIP score is found”
Rethinking FID: Towards a Better Evaluation Metric for Image Generation (2023, methodology): “Previous work pointed to unreliability of evaluation metrics in image generation [5, 17].”
Lumiere: A Space-Time Diffusion Model for Video Generation (2024, unknown): “The conditioning signal C contains this first frame followed by blank frames for the rest of the video.”
MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, methodology): “To quantify these observations, we use the Frechet Inception Distance (FID) (Parmar et al., 2022) to measure the distance between the distribution of 512× 512 crops from the panoramic images to the distribution of images generated by the reference model Φ.”