Classification Accuracy Score for Conditional Generative Models

By Suman V. Ravuri, Oriol Vinyals. 2019

272 citations recorded by the source corpus.

Deep generative models (DGMs) of images are now sufficiently mature that they produce nearly photorealistic samples and obtain scores similar to the data distribution on heuristics such as Frechet Inception Distance (FID). These results, especially on large-scale datasets such as ImageNet, suggest that DGMs are learning the data distribution in a perceptually meaningful space and can be used in downstream tasks. To test this latter hypothesis, we use class-conditional generative models from a number of model classes---variational autoencoders, autoregressive models, and generative adversarial networks (GANs)---to infer the class labels of real data. We perform this inference by training an image classifier using only synthetic data and using the classifier to predict labels on real data. The performance on this task, which we call Classification Accuracy Score (CAS), reveals some surprising results not identified by traditional metrics and constitute our contributions. First, when using a state-of-the-art GAN (BigGAN-deep), Top-1 and Top-5 accuracy decrease by 27.9\% and 41.6\%, respectively, compared to the original data; and conditional generative models from other model classes, such as Vector-Quantized Variational Autoencoder-2 (VQ-VAE-2) and Hierarchical Autoregressive Models (HAMs), substantially outperform GANs on this benchmark. Second, CAS automatically surfaces particular classes for which generative models failed to capture the data distribution, and were previously unknown in the literature. Third, we find traditional GAN metrics such as Inception Score (IS) and FID neither predictive of CAS nor useful when evaluating non-GAN models. Furthermore, in order to facilitate better diagnoses of generative models, we open-source the proposed metric.

Subfields: Computer Vision and Pattern Recognition, General Social Sciences, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In General Social Sciences

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Similarly, Ravuri & Vinyals (2019) show that models with high sample quality fail to generate data from which an accurate classiﬁer can be trained.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “GANs have also been used to generate training data directly [37, 33, 56, 1, 44], however this approach does not seem to be as beneﬁcial as learning sequences of data augmentation operations that are pre-deﬁned [40].”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “We also evaluate our method using the recently proposed Classification Accuracy Score (CAS) [28], which requires training an ImageNet classifier only on samples from the candidate model, but then evaluates its classification accuracy on real images from the test set, thus measuring sample quality an”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, background): “, 2017) and classification accuracy score (Ravuri and Vinyals, 2019), the latter by a large margin.”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “Commonly-used ones include log-likelihood [28, 41], density estimate with Parzen window [28], Inception Score [64], Perceptual Path Length [37], Fréchet Inception Distance (FID) [30], Classification Accuracy Score and its early variants [60, 64], Classifier Twosample Tests [44, 47], precision and re”
Image Super-Resolution via Iterative Refinement (2021, background): “instability and mode collapse [19], [20], [21], [22].”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “As generative models continue to improve, there is growing interest in their use as a source of cheap and plentiful training data for downstream tasks [32, 46, 49, 57, 64, 65].”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “We also report CAS evaluated on classiﬁers trained without augmentation from RandAugment [9].”
Palette: Image-to-Image Diffusion Models (2021, neutral): “C V ] 3 M ay 2 02 2 Ravuri and Vinyals 2019].”
Image Synthesis From Reconfigurable Layout and Style (2019, methodology): “• It shows state-of-the-art performance in terms of the inception score [30], Frèchet Inception Distance [9], diversity score [37] and classification accuracy [28] on two widely used datasets, the COCO-Stuff [2] and the Visual Genome [20].”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Moreover, performance drops when even relatively small amounts of synthetic data are added to real data during training [45].”
StyleGAN2 Distillation for Feed-forward Image Manipulation (2020, background): “[44] study the quality of a classificator trained on synthetic data generated by BigGAN and show [45] that BigGAN does not capture the ImageNet [14] data distributions and is only partly successful for data augmentation.”
Unsupervised anomaly localization using VAE and beta-VAE (2020, background): “Recent works such as [20] propose substantial upgrades, however other works such as [17] still supports that GANs have more trouble than other generative models to cover the whole distribution support.”
Automated curricula through setter-solver interactions (2019, background): “In particular, while conditioning directly on an informative latent variable can work well, for example when trying to generate images from a given class (Mirza & Osindero, 2014; Brock et al., 2018), even this problem is not completely solved (Ravuri & Vinyals, 2019).”
Generation and evaluation of synthetic patient data (2020, background): “Additionally, works such as [55] have reported that while GANs often produce high quality synthetic data (for example realistic looking synthetic images), with respect to utility metrics such as classification accuracy they often underperform compared to likelihood based models.”