On the quantitative analysis of deep belief networks

By Ruslan Salakhutdinov, Iain Murray. 2008

525 citations recorded by the source corpus.

Deep Belief Networks (DBN's) are generative models that contain many layers of hidden variables. Efficient greedy algorithms for learning and approximate inference have allowed these models to be applied successfully in many application domains. The main building block of a DBN is a bipartite undirected graphical model called a restricted Boltzmann machine (RBM). Due to the presence of the partition function, model selection, complexity control, and exact maximum likelihood learning in RBM's are intractable. We show that Annealed Importance Sampling (AIS) can be used to efficiently estimate the partition function of an RBM, and we present a novel AIS scheme for comparing RBM's with different architectures. We further show how an AIS estimator, along with approximate inference, can be used to estimate a lower bound on the log-probability that a DBN model with multiple hidden layers assigns to the test data. This is, to our knowledge, the first step towards obtaining quantitative results that would allow us to directly assess the performance of Deep Belief Networks as generative models of data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Variational Inference and Learning in Belief Networks (2014, methodology): “The fDARN and multi-layer SBN models trained using NVIL also out-perform a 500-hidden-unit RBM trained with 3-step contrastive divergence (CD), but not the one trained with 25-step CD (Salakhutdinov & Murray, 2008).”
Deep AutoRegressive Networks (2013, methodology): “DARN with 500 stochastic hidden units compares favourably to state-of-the-art generative performance of deep Boltzmann machines (DBM; Salakhut-dinov & Hinton, 2009) and deep belief networks (DBN; Salakhutdinov & Murray, 2008; Murray & Salakhutdinov, 2009).”
Neural Autoregressive Distribution Estimation (2016, methodology): “We now consider the case of an image dataset, constructed by binarizing the MNIST digit dataset, as generated by Salakhutdinov and Murray (2008).”
MADE: Masked Autoencoder for Distribution Estimation (2015, methodology): “…research is to test the common assumption that the use of models with probabilistic latent variables and intractable partition functions (such as the restricted Boltz-mann machine (Salakhutdinov & Murray, 2008)), is a necessary evil in designing powerful generative models for high-dimensional data.”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “MNIST digits can be either statically or dynamically binarized — for the statically binarized dataset we used the binarization from [60], and for the dynamically binarized dataset we sampled images from Bernoulli distributions with probabilities equal to the continuous values of the images in the…”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “Table 2: Test log-likelihood on statically bina-rized MNIST for different number of importance weighted samples.”
NICE: Non-linear Independent Components Estimation (2014, background): “In addition, the log-likelihood is intractable, and the best known estimation procedure, annealed importance sampling (AIS) (Salakhutdinov and Murray, 2008), might yield an overly optimistic evaluation (Roger Grosse and Salakhutdinov, 2013).”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, methodology): “It was previously observed that one of the potential failure modes of Boltzmann machines is to fail to generate one or more modes of a distribution or to drastically misallocate probability mass between modes [24].”
Categorical Reparameterization with Gumbel-Softmax (2016, methodology): “We use the MNIST dataset with ﬁxed binarization for training and evaluation, which is common practice for evaluating stochastic gradient estimators (Salakhutdinov & Murray, 2008; Larochelle & Murray, 2011).”
Importance Weighted Autoencoders (2015, background): “This has the advantage that many of the conditional distributions are tractable, but the inability to sample from the model or compute the partition function has been a major roadblock (Salakhutdinov & Murray, 2008).”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, methodology): “For MNIST we used the fixed binarization of Salakhutdinov & Murray (2008) and the standard 50,000/10,000/10,000 split into training/validation/testing sets.”
RNADE: The real-valued neural autoregressive density-estimator (2013, methodology): “Restricted Boltzmann Machines (RBMs), which are undirected graphical models, fit samples of binary vectors from a range of sources better than mixture models [4, 5].”
PixelVAE: A Latent Variable Model for Natural Images (2016, methodology): “We evaluate our model on the binarized MNIST dataset (Salakhutdinov & Murray, 2008; Lecun et al., 1998) and report results in Table 1.”
A Deep and Tractable Density Estimator (2013, methodology): “…method with mixtures of multivariate Bernoulli distributions with 10 and 500 components, ﬁxed-ordering NADEs, RBMs (500 hidden units), and two-hidden-layer DBNs (500 and 2000 hidden units on each layer) whose performance was estimated by Salakhutdinov & Murray (2008); Murray & Salakhutdinov (2009).”
One-Shot Generalization in Deep Generative Models (2016, background): “Salakhutdinov et al. (2013) developed a probabilistic model that combines a deep Boltzmann machine with a hierarchical Dirichlet process to learn hierarchies of concept categories as well as provide a powerful generative model.”
Deep Gaussian Processes (2012, background): “A significant amount of work has then to be done with annealed importance sampling if even the likelihood1 of a data set under the RBM model is to be estimated [Salakhutdinov and Murray, 2008].”