Bounding the Test Log-Likelihood of Generative Models

By Yoshua Bengio, L. Yao. 2013

23 citations recorded by the source corpus.

Several interesting generative learning algorithms involve a complex probability distribution over many random variables, involving intractable normalization constants or latent variable normalization. Some of them may even not have an analytic expression for the unnormalized probability function and no tractable approximation. This makes it difficult to estimate the quality of these models, once they have been trained, or to monitor their quality (e.g. for early stopping) while training. A previously proposed method is based on constructing a non-parametric density estimator of the model's probability function from samples generated by the model. We revisit this idea, propose a more efficient estimator, and prove that it provides a lower bound on the true test log-likelihood, and an unbiased estimator as the number of generated samples goes to infinity, although one that incorporates the effect of poor mixing. We further propose a biased variant of the estimator that can be used reliably with a finite number of samples for the purpose of model comparison.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Generalized Denoising Auto-Encoders as Generative Models (2013, methodology): “-parametric density estimator P^(x) = mean ~ X P(xjX~) constructed from 10000 consecutively generated samples (X~ from the Markov chain). The expected value of E[P^(x)] over the samples can be shown (Bengio and Yao, 2013) to be a lower bound (i.e. conservative estimate) of the true (implicit) model ”
GSNs : Generative Stochastic Networks (2015, methodology): “The following theorem is proved by Bengio et al. (2013c).”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, methodology): “Kernel density estimation (KDE) (Parzen, 1962) is widely used to evaluate decoder-based models (Goodfellow et al., 2014; Li & Swersky, 2015), and a variant was proposed in the setting of evaluating Boltzmann machines (Bengio et al., 2013).”
Revisiting Classifier Two-Sample Tests (2016, methodology): “y the generative model. The use of two-sample tests to evaluate the sample quality of generative models include the pioneering work of Box (1980), the use of Maximum Mean Discrepancy (MMD) criterion (Bengio et al., 2013; Dziugaite et al., 2015; Lloyd & Ghahramani, 2015; Bounliphone et al., 2015;”
Accurate and conservative estimates of MRF log-likelihood using reverse annealing (2014, methodology): “Bengio et al. (2013) highlighted the problem of optimistic RBM log-likelihood estimates and proposed a pessimistic estimator based on nonparametric density estimation.”
Multimodal Transitions for Generative Stochastic Networks (2013, methodology): “The CSL estimates are computed on different numbers of consecutive samples, all starting from a random intialization, as opposed to computing the estimates based on samples collected every 100 steps originally reported in [9].”