A note on the evaluation of generative models

By Lucas Theis, Aäron van den Oord, Matthias Bethge. 2015

1,233 citations recorded by the source corpus.

Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

MaskGAN: Better Text Generation via Filling in the ______ (2018, background): “Theis et al. (2016) also shows how validation perplexity does not necessarily correlate with sample quality.”
Pixel Recurrent Neural Networks (2016, background): “When uniform noise is added (with values in the interval [0, 1]), then the log-likelihoods of continuous and discrete models are directly comparable (Theis et al., 2015).”
Challenges in Detoxifying Language Models (2021, positive): “The empirical and theoretical results above, along with those in Theis et al. (2015) and van den Oord and Dambre (2015), provide several examples where small loss gaps can miss important biases.”
Are GANs Created Equal? A Large-Scale Study (2017, background): “[25] provide an analysis of common failure modes and demonstrate that it is possible to achieve high likelihood, but low visual quality, and vice-versa.”
Evaluating Lossy Compression Rates of Deep Generative Models (2020, methodology): “This problem was observed by Theis et al. (2015) in the context of log-likelihood estimation; our current observation indicates that rate distortion analysis does not ﬁx this problem.”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, background): “First, NLL in pixel space is not always a good measure of sample quality [37], and cannot be reliably used to make comparisons between different model classes.”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, background): “…et al. (2015) note that training a continuous density model p model on uniformly dequantized data y can be interpreted as maximizing a lower bound on the log-likelihood for a certain discrete model P model on the original discrete data x : The argument of Theis et al. (2015) proceeds as follows.”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, background): “Objectively evaluating implicit generative models is difﬁcult (Theis et al., 2015).”
Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “It has been argued (Theis et al., 2015; Bousquet et al., 2017; Ar-jovsky et al., 2017; Bellemare et al., 2017) that the KL may not always be appropriate from this perspective, by making sub-optimal trade-offs between likelihood and similarity.”
Generalizing Hamiltonian Monte Carlo with Neural Networks (2017, neutral): “Although higher log likelihood does not necessarily correspond to better samples (Theis et al., 2015), we can see in Figure 5, shown in the Appendix, that the decoder trained with L2HMC generates sharper samples than the compared methods.”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, methodology): “The MNIST, SVHN and CIFAR10 datasets were used as is except for uniform noise added to MNIST and CIFAR10, as per Theis et al. (2016), and the aligned and cropped version of CelebA was scaled from 218 x 178 pixels to 78 x 64 pixels and center-cropped at 64 x 64 pixels (Liu et al., 2015).”
Learning to Generate Samples from Noise through Infusion Training (2017, methodology): “Since most previous published results on non-likelihood based models (such as GANs) used a Parzen-window-based estimator (Breuleux et al., 2011), we use it as our ﬁrst comparison tool, even if it can be misleading (Lucas Theis & Bethge, 2016).”
Conditional Image Generation with PixelCNN Decoders (2016, background): “On the other hand, as noted in [27], we observed great improvements in the visual quality of the generated samples.”
Density estimation using Real NVP (2016, background): “For examples, the pixel values of an image typically lie in [0, 256] after application of the recommended jittering procedure [64, 62].”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, methodology): “The log likelihood bounds reported here are instead for data that has been pre-processed by adding uniform noise to remove pixel quantization, as recommended in (Theis et al., 2015).”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “Each metric has associated pros and cons [6, 72] and none are perfect.”
Diagnosing and Enhancing VAE Models (2019, background): “While impressive log-likelihood gains have been demonstrated, this achievement is largely orthogonal to the goal of improving quantitative measures of visual quality (Theis et al., 2016), which has been our focus herein.”
Unrolled Generative Adversarial Networks (2016, background): “Additionally, because it is intractable to compute the GAN training loss, and because approximate measures of performance such as Parzen window estimates suffer from major flaws (Theis et al., 2016), evaluation of GAN performance is challenging.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Additionally, Theis et al. (2016) discussed potential drawbacks of forward KL divergence in generative models and Huszar (2015) proposed a generalization of Jensen-Shannon divergence that interpolates between KL and reverse KL and has Jensen-Shannon as its midpoint.”