Learning Deep Boltzmann Machines using Adaptive MCMC

By Ruslan Salakhutdinov. 2010

85 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Better Mixing via Deep Representations (2012, methodology): “To improve mixing, a powerful idea that has been explored recently for deep learning algorithms (Desjardins et al. , 2010; Cho et al. , 2010; Salakhutdinov, 2010b,a) is tempering (Neal, 1994).”
Neural Autoregressive Distribution Estimation (2016, methodology): “Another line of development has been to optimize likelihood using RobbinsMonro stochastic approximation (Younes, 1989), also known as Persistent CD (Tieleman, 2008), and develop good MCMC samplers for deep undirected models (Salakhutdinov, 2009, 2010; Desjardins et al., 2010; Cho et al., 2010).”
Representation Learning: A Review and New Perspectives (2012, methodology): “For RBMs and some (not too deep) Boltzmann machines, one option is the use of Annealed Importance Sampling (Murray and Salakhutdinov, 2009) in order to estimate the partition function (and thus the test log-likelihood).”
Deep Learning of Representations: Looking Forward (2013, background): “nd then gradually cool to the target distribution while continuing to make MCMC moves, to make sure we end up in one of the “islands” of high probability. Desjardins et al. (2010); Cho et al. (2010); Salakhutdinov (2010b,a) have all considered various forms of tempering to address the failure of Gib”
On Tracking The Partition Function (2011, methodology): “In our experiments, we avoid this issue by using the intermediate distributions q i We tested mini-batch sizes N ∈ [10 , 20] .”
A Two-Stage Pretraining Algorithm for Deep Boltzmann Machines (2013, methodology): “We used the MNIST and Caltech101 Silhouettes datasets because experimental results of using DBMs for both datasets are readily available for direct comparison [13,12,9].”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, background): “Tieleman and Hinton (2009) have proposed quite a different approach to addressing potential mixing problems of SML with their fast-weights persistent contrastive divergence (FPCD), and it has also been exploited to train Deep Boltz-mann Machines (Salakhutdinov, 2010a) and construct a pure sampling a”
Deep Logic Networks: Inserting and Extracting Knowledge From Deep Belief Networks (2018, unknown): “The common way to avoid the above intractability is to learn a deep network layer by layer [3], [24], [27].”
Optimizing Neural Networks that Generate Iimages (2014, unknown): “(Salakhutdinov, 2010) voices the same concern.”
An Efficient Learning Procedure for Deep Boltzmann Machines (2012, methodology): “97, achieved by a carefully trained two-hidden-layer Deep Belief Network (Salakhutdinov and Murray (2008)).”
On layer-wise representations in deep neural networks (2013, background): “The importance of producing data-dependent statistics from a converged Gibbs sampler, or an accurate approximation of it, was already demonstrated by Salakhutdinov and Larochelle (2010). Similarly, low variance is also important for modeling both data-dependent and data-independent statistics.”
Foundations and Advances in Deep Learning (2014, background): “Similarly, the tempered transition and the coupled adaptive simulated tempering (Salakhutdinov, 2010) are all based on using tempered distributions.”