Accurate and conservative estimates of MRF log-likelihood using reverse annealing

By Yuri Burda, Roger Baker Grosse, Ruslan Salakhutdinov. 2014

67 citations recorded by the source corpus.

Markov random fields (MRFs) are difficult to evaluate as generative models because computing the test log-probabilities requires the intractable partition function. Annealed importance sampling (AIS) is widely used to estimate MRF partition functions, and often yields quite accurate results. However, AIS is prone to overestimate the log-likelihood with little indication that anything is wrong. We present the Reverse AIS Estimator (RAISE), a stochastic lower bound on the log-likelihood of an approximation to the original MRF model. RAISE requires only the same MCMC transition operators as standard AIS. Experimental results indicate that RAISE agrees closely with AIS log-probability estimates for RBMs, DBMs, and DBNs, but typically errs on the side of underestimating, rather than overestimating, the log-likelihood.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “Trained models are reported with 1 importance weighted sample, L1, and 1000 importance weighted samples, L1e3 [3].”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, background): “…can write the ratio of probabilities in this term as, p ( s 0 ) π T 1 ( s 0 ) π which is similar in shape (but arising in a different context) to the product of probability ratios computed for annealed importance sampling (Neal, 2001) and reverse annealed importance sampling (Burda et al., 2014).”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “In (Burda et al., 2014) it is shown that AIS can also be performed using the reverse rather than forward trajectory.”
Importance Weighted Autoencoders (2015, methodology): “For the OMNIGLOT dataset, the best performing IWAE has log-likelihood of -103.38 nats, which is slightly worse than the log-likelihood of -100.46 nats achieved by a Restricted Boltzmann Machine with 500 hidden units trained with persistent contrastive divergence (Burda et al., 2015).”
Variational Lossy Autoencoder (2016, neutral): “(Salimans et al., 2014) 81.94 DLGM 2hl + IWAE (Burda et al., 2015a) 82.90 Discrete VAE (Rolfe, 2016) 80.04 LVAE (Kaae Sønderby et al., 2016) 81.74 DRAW + VGP (Tran et al., 2015) 79.88 IAF VAE (Kingma et al., 2016) 79 As seen in Table 2,3,4, with the same set of hyperparameters tuned on statically ”
Discrete Variational Autoencoders (2016, background): “…to the importance-weighted autoencoder (IWAE; Burda et al., 2016); ladder variational autoencoder (Ladder VAE; Sønderby et al., 2016); and the restricted Boltzmann machine (RBM; Smolensky, 1986) and deep belief network (DBN; Hinton et al., 2006), reporting the results of Burda et al. (2015).”
Implicit Generation and Generalization in Energy-Based Models (2019, methodology): “To evaluate the likelihood of EBMs, we use AIS (Neal, 2001) and RAISE to obtain a lower bound of partition function (Burda et al., 2015).”
Semi-Supervised Generation with Cluster-aware Generative Models (2017, methodology): “Inactive stochastic units A common problem encountered when training VAEs with bottom-up inference networks is given by the so called inactive units in the higher layers of stochastic variables (Burda et al., 2015a; Sønderby et al., 2016).”
Sandwiching the marginal likelihood using bidirectional Monte Carlo (2015, methodology): “Our BDMC method is similar in spirit to the reverse AIS estimator (Burda et al., 2015), which has been used to evaluate log-likelihoods of Markov random ﬁelds.”
Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix (2015, methodology): “We computed 95% confidence intervals using the bootstrap, similarly to Burda et al. (2014).”
Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces (2019, methodology): “In Table 1, We report result from Reverse AIS as it tends to over-estimate the partition function thus underestimate the density.”
Implicit Generation and Modeling with Energy Based Models (2019, methodology): “To evaluate the likelihood of EBMs, we use AIS [Neal, 2001] and RAISE to obtain a lower bound of partition function [Burda et al., 2015].”
Bidirectional Helmholtz Machines (2015, background): “r estimators for log ^p(x) and log ~p(x) report numbers that differ by multiple nats (for MNIST and TFD experiments). A ﬁrst attempt to estimate Zcould certainly be made by applying AIS [20] or RAISE [21]. Another direction for future research could involve semisupervised learning: The symmetric nat”
Filtering Variational Objectives (2017, methodology): “[28, 29] use Jensen’s inequality in a forward and reverse estimator to detect the failure of inference methods.”
Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching (2019, methodology): “For energy-based models, the log density can be obtained after estimating the partition function with Annealed Importance Sampling (AIS) (Salakhutdinov and Murray, 2008) or Reverse AIS (Burda et al., 2015). In our experiment on CIFAR-10 model, similar to reports in Du and Mordatch (2019), there is s”
On Fenchel Mini-Max Learning (2019, background): “The inability to take infinite samples makes unbiased estimation of unnormalized statistical models a long-standing challenge posed to the statistical community, especially for high-dimensional problems [9].”