Sandwiching the marginal likelihood using bidirectional Monte Carlo

By Roger Baker Grosse, Zoubin Ghahramani, Ryan P. Adams. 2015

64 citations recorded by the source corpus.

Computing the marginal likelihood (ML) of a model requires marginalizing out all of the parameters and latent variables, a difficult high-dimensional summation or integration problem. To make matters worse, it is often hard to measure the accuracy of one's ML estimates. We present bidirectional Monte Carlo, a technique for obtaining accurate log-ML estimates on data simulated from a model. This method obtains stochastic lower bounds on the log-ML using annealed importance sampling or sequential Monte Carlo, and obtains stochastic upper bounds by running these same algorithms in reverse starting from an exact posterior sample. The true value can be sandwiched between these two stochastic bounds with high probability. Using the ground truth log-ML estimates obtained from our method, we quantitatively evaluate a wide variety of existing ML estimators on several latent variable models: clustering, a low rank approximation, and a binary attributes model. These experiments yield insights into how to accurately estimate marginal likelihoods.

Subfields: Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Evaluating Lossy Compression Rates of Deep Generative Models (2020, background): “The combination of the AIS lower bound and upper bound on the log partition function is called bidirectional Monte Carlo (BDMC), and the gap between these bounds is called the BDMC gap (Grosse et al., 2015).”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, methodology): “In particular, log p̂(x) is a stochastic lower bound on log p(x), satisfying E[log p̂(x)] ≤ log p(x) and Pr(log p̂(x) > log p(x) + b) < e−b. Grosse et al. (2015) pointed out that if AIS is run in reverse starting from an exact posterior sample, it yields an unbiased estimate of 1/p(x), which (by…”
Gaussian Process Behaviour in Wide Deep Neural Networks (2018, background): “The value of the log-marginal likelihood computed in the two diﬀerent ways agree to within a single nat which is negligible from a model selection perspective (Grosse et al., 2015).”
Inference Suboptimality in Variational Autoencoders (2018, methodology): “This is due to the backward chain initializes from exact posterior samples (Grosse et al., 2015).”
Filtering Variational Objectives (2017, methodology): “[28, 29] use Jensen’s inequality in a forward and reverse estimator to detect the failure of inference methods.”
Importance Weighted Hierarchical Variational Inference (2019, methodology): “The proposed upper bound provides a variational alternative to MCMC-based upper bounds (Grosse et al., 2015) and complements the standard Importance Weighted stochastic lower bound of Burda et al. (2015) on the log marginal density: LK = E τ(ψ1:K |z,x) log ( 1 K K∑ k=1 q(z, ψk | x) τ(ψk | z, x) ) ≤…”
Adversarial Message Passing For Graphical Models (2016, methodology): “This procedure yields two Markov chains transitioning from evidence to prior and from prior to evidence in a setting similar to that used for the Bennet acceptance ratio estimator [16] and related newer work [17, 18, 19, 20, 21].”