A Note on the Inception Score

By Shane T. Barratt, Rishi Sharma. 2018

822 citations recorded by the source corpus.

Deep generative models are powerful tools that have produced impressive results in recent years. These advances have been for the most part empirically driven, making it essential that we use high quality evaluation metrics. In this paper, we provide new insights into the Inception Score, a recently proposed and widely used evaluation metric for generative models, and demonstrate that it fails to provide useful guidance when comparing models. We discuss both suboptimalities of the metric itself and issues with its application. Finally, we call for researchers to be more systematic and careful when evaluating and comparing generative models, as the advancement of the field depends upon it.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

Improved Techniques for Training Score-Based Generative Models (2020, background): “However, we note that Inception and FID scores have known issues [21, 22] and they should be interpreted with caution as they may not correlate with visual quality in the expected way.”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “For a more detailed discussion on the variability of IS, please refer to Barratt & Sharma (2018).”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, neutral): “Noting that there are several known drawbacks to these metrics [3, 32, 19], we report our results in Fig.”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, background): “Table 1 (a) compares the Inception Score (IS) [41, 3] and Fréchet Inception Distance (FID) [17] with Inception v3 classifier [43] on 40,000 generated examples.”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, background): “Of these, the Inception Score (IS, Salimans et al. (2016)) and Fr ´ echet Inception Distance (FID, Heusel et al. (2017)) have become popular despite their notable ﬂaws (Barratt & Sharma, 2018).”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “We do not use IS [49] as it has been noted to be insufﬁcient for model evaluation [2].”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “One drawback of this metric is that it does not reward covering the whole distribution or capturing diversity within a class, and models which memorize a small subset of the full dataset will still have high IS [3].”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “Note that FID reflects the sample quality more faithfully, as potential problems have been reported for IS on CIFAR-10 (Barratt & Sharma, 2018).”
Object-Centric Image Generation from Layouts (2020, background): “The Inception Score has been criticised (Barratt and Sharma 2018), notably due to issues caused by the mismatch between the domain it was trained on (the ImageNet dataset comprising single objects of interest) and the domain of VG and COCO-Stuff images (comprising multiple objects in complex scenes)”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, background): “ human evaluation in assessing the realism and variation of the generated images [16], while IS is limited by what the Inception classiﬁer can recognise, which is directly linked to its training data [3]. If one learns to generate something not present in the classiﬁers training data (e.g. human fac”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “While IS and FID have both been shown to correlate with human judgements of generated image quality, IS is likely less informative as it overfits easily and can be manipulated to achieve much higher scores using simple tricks [2, 17].”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “Since it has some flaws Barratt and Sharma [31], we use FID as the main metric in this work.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “Yet these measures are not aligned with our intentions: they are a poor proxy for the actual performance and produce degenerate solutions when optimized directly (Barratt & Sharma, 2018).”
Assessing Generative Models via Precision and Recall (2018, background): “IS necessitates a labeled data set and has been found to be weak at providing guidance for model comparison [3].”
Classification Accuracy Score for Conditional Generative Models (2019, background): “One can easily diagnose the issue with IS: as noted in [28], IS does not account for intra-class diversity, and a training set with little intra-class diversity may make the classifier fail to generalize to a more diverse test set.”
BooVAE: Boosting Approach for Continual Learning of VAE (2019, methodology): “FID rely heavily on the implementation of Inception network (Barratt & Sharma, 2018); we use PyTorch version of the Inception V3 network (Paszke et al., 2017).”
Generative Models from the perspective of Continual Learning (2018, background): “to improve on the Inception Score (IS) [24] which has many intrinsic shortcomings, as well as additional problems when used on a dataset different than ImageNet [25].”