The Role of ImageNet Classes in Fréchet Inception Distance

By Tuomas Kynkaanniemi, Tero Karras, Miika Aittala, Timo Aila, Jaakko Lehtinen. International Conference on Learning Representations, 2022

295 citations recorded by the source corpus.

Fr\'echet Inception Distance (FID) is the primary metric for ranking models in data-driven generative modeling. While remarkably successful, the metric is known to sometimes disagree with human judgement. We investigate a root cause of these discrepancies, and visualize what FID"looks at"in generated images. We show that the feature space that FID is (typically) computed in is so close to the ImageNet classifications that aligning the histograms of Top-$N$ classifications between sets of generated and real images can reduce FID substantially -- without actually improving the quality of results. Thus, we conclude that FID is prone to intentional or accidental distortions. As a practical example of an accidental distortion, we discuss a case where an ImageNet pre-trained FastGAN achieves a FID comparable to StyleGAN2, while being worse in terms of human evaluation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety Research

View Paper: https://www.semanticscholar.org/paper/75348e1f3c800fcdce2d4d64b5a6aa3c69a1fb7d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety Research

Stefano Ermon

Citations in the corpus

Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, methodology): “In Table 3, we show that our baseline model achieves a new state-of-the-art CLIP-FID score [24] of 10.”
GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models (2023, background): “Measuring the quality of generated image has been a challenging and ongoing research problem in computer vision [6, 38, 44].”
Improved Techniques for Training Consistency Models (2023, background): “As noted by Kynkäänniemi et al. (2023), this can skew FIDs and lead to inflated sample quality.”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, methodology): “As per [30], we use the features extracted from the CLIP model to compute FID, which we find better correlates the perceptual quality.”
Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “, CLIP-FID (Kynkäänniemi et al. 2023), which uses the CLIP (Radford et al.”
Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion (2023, result): “Although existing literature (Kynkäänniemi et al., 2023) discusses the possibility that FID improvement may not necessarily correlate with an actual enhancement in human perceptual judgement, our observations in Figure 14 indicates that, in the case of CTM, the improvement achieved through GAN is…”
Pretraining is All You Need for Image-to-Image Translation (2022, methodology): “Following [27], we calculate FID [18] with the CLIP model [39] whose feature space is more robust and transferable.”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “By involving these irregular masks into training, our model is able to generate photo-realistic results given various shape masks.”