On the Quantitative Analysis of Decoder-Based Generative Models

By Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, Roger Baker Grosse. 2016

226 citations recorded by the source corpus.

The past several years have seen remarkable progress in generative models which produce convincing samples of images and other modalities. A shared component of many powerful generative models is a decoder network, a parametric deep neural net that defines a generative distribution. Examples include variational autoencoders, generative adversarial networks, and generative moment matching networks. Unfortunately, it can be difficult to quantify the performance of these models because of the intractability of log-likelihood estimation, and inspecting samples can be misleading. We propose to use Annealed Importance Sampling for evaluating log-likelihoods for decoder-based models and validate its accuracy using bidirectional Monte Carlo. The evaluation code is provided at this https URL. Using this technique, we analyze the performance of decoder-based models, the effectiveness of existing log-likelihood estimators, the degree of overfitting, and the degree to which these models miss important modes of the data distribution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

In Signal Processing

Citations in the corpus

Are GANs Created Equal? A Large-Scale Study (2017, methodology): “[26] suggest an annealed importance sampling algorithm to estimate the hold-out log-likelihood.”
Evaluating Lossy Compression Rates of Deep Generative Models (2020, methodology): “For 100 dimensional models, we used N = 1600000 intermediate distributions in order to obtain small BDMC gaps.”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, background): “A variety of works have proposed heuristics for measuring the sample quality of models without tractable likelihoods (Salimans et al., 2016; Heusel et al., 2017; Bińkowski et al., 2018; Wu et al., 2017).”
Learning to Generate Samples from Noise through Infusion Training (2017, methodology): “In Table 2 we compare our model with the recent Annealed Importance Sampling results (Wu et al., 2016).”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “[32] observe that the marginal loglikelihood, estimated by non-encoder-based methods, is not sensitive to the encoder overfitting (see also Fig.”
Comparison of Maximum Likelihood and GAN-based training of Real NVPs (2017, background): “Ap-proximating the likelihood by annealed importance sampling (Wu et al., 2016) would not discover this problem, as their analysis assumes a Gaussian observation model with a ﬁxed variance.”
Wasserstein GAN (2017, background): “For example, we can see in the recent paper [23] that the optimal standard deviation of the noise added to the model when maximizing likelihood is around 0.1 to each pixel in a generated image, when the pixels were already normalized to be in the range [0 , 1].”
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, methodology): “Even though the results suggest that AVB with AC can also overestimate the true ELBO in higher dimensions, we note that the loglikelihood estimate computed by AIS is also only a lower bound to the true log-likelihood (Wu et al., 2016).”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “Wu et al. (2017) state that it is di ﬃ cult to quantify GAN performance, and propose Monte Carlo methods to approximate the log-likelihood of GANs on MNIST images.”
Assessing Generative Models via Precision and Recall (2018, background): “While the likelihood can be approximated in some settings, kernel density estimation in high-dimensional spaces is extremely challenging [22, 24].”
Brain-inspired replay for continual learning with artificial neural networks (2020, neutral): “|x) [logpχ(z|y)] = Ez∼N ( μ(x),σ(x) 2 I ) [logN (z|μy,σy2I)] = Ez∼N ( μ(x),σ(x) 2 I ) [−1 2(z−μy)T (σy2I)−1 (z−μ)− log (√ (2π)J |σy2I| )] = Ez∼N ( μ(x),σ(x) 2 I ) −1 2 J ∑ j=1 ( zj−μyj )2 σ j 2 − (1)2 log (2π)J J ∏ j=1 σ j 2  = E ∼N (0,I) −1 2 J ∑ j=1 ( μ (x) j +σ (x) j j−μ y j )2 σ j 2 −”
Learning in Implicit Generative Models (2016, methodology): “…reporting the value of the density ratio, a quantity we can always compute using the approaches described, the use of annealed importance sampling Wu et al. (2016), use of empirical distance metric such as the MMD (Sutherland et al., 2016) or the Wasserstein distance (Arjovsky et al., 2017).”
LOGAN: Membership Inference Attacks Against Generative Models (2017, background): “refore it is more difﬁcult to detect overﬁtting and mount the attack. In fact, detecting overﬁtting in generative models is regarded as one of the most important research problems in machine learning [54]. Also, model inversion attacks allow adversaries to extract training data from outputted model ”