Tempered Markov Chain Monte Carlo for training of Restricted Boltzmann Machines

By Guillaume Desjardins, Aaron C. Courville, Yoshua Bengio, Pascal Vincent, Olivier Delalleau. 2010

122 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Better Mixing via Deep Representations (2012, methodology): “To improve mixing, a powerful idea that has been explored recently for deep learning algorithms (Desjardins et al., 2010; Cho et al., 2010; Salakhutdinov, 2010b,a) is tempering (Neal, 1994).”
Neural Autoregressive Distribution Estimation (2016, methodology): “Another line of development has been to optimize likelihood using RobbinsMonro stochastic approximation (Younes, 1989), also known as Persistent CD (Tieleman, 2008), and develop good MCMC samplers for deep undirected models (Salakhutdinov, 2009, 2010; Desjardins et al., 2010; Cho et al., 2010).”
Representation Learning: A Review and New Perspectives (2012, background): “Desjardins et al. (2010); Cho et al. (2010); Salakhutdinov (2010b,a) have all considered various forms of tempered transitions to address the failure of Gibbs chain mixing, and convincing solutions have not yet been clearly demonstrated.”
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (2019, methodology): “is Markov-chain Monte-Carlo (MCMC) sampling (Neal, 1993; Swendsen and Wang, 1986; Salakhutdinov, 2009; Desjardins et al., 2010; Cho et al., 2010).”
Deep Learning of Representations: Looking Forward (2013, background): “Desjardins et al. (2010); Cho et al. (2010); Salakhutdinov (2010b,a) have all considered various forms of tempering to address the failure of Gibbs chain mixing in RBMs.”
Learning Deep Boltzmann Machines using Adaptive MCMC (2010, background): “When learning Restricted Boltzmann Machines (RBM’s), (Desjardins et al., 2010; Salakhutdinov, 2010) show that this can often lead to poor generative models.”
Learning protein constitutive motifs from sequence data (2018, methodology): “We also used occasionnally Parallel Tempering for training and for sequence design to improve thermalization speed and sampling accuracy, see [57, 60] and Supplemental Material of [46] for an introduction.”
Training Restricted Boltzmann Machines (2015, background): “This can prevent the likelihood from diverging [5].”
A topological insight into restricted Boltzmann machines (2016, background): “…(e.g. persistent contrastive divergence (Tieleman, 2008), fast persistent contrastive divergence (Tieleman and Hinton, 2009), parallel tempering (Desjardins et al, 2010)), or to replace the Gibbs sampling with a transition operator for a faster mixing rate and to improve the learning accuracy…”
On Tracking The Partition Function (2011, methodology): “However, from the perspective of tracking the log partition function, we will see in Section 3 that the SML-PT scheme [7] presents a rather unique advantage.”
On the Expressive Power of Deep Architectures (2011, background): “It turned out that SML yields RBMs with much better likelihood, whereas CD updates sometimes give rise to worsening likelihood and suffer from other issues (Desjardins et al., 2010).”
Parallel tempering is efficient for learning restricted Boltzmann machines (2010, background): “Recently, the use of PT learning for RBMs has been proposed independently also in [ 14 ].”
Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines (2011, methodology): “The use of advanced MCMC sampling methods such as PT has been shown to avoid divergence but the likelihood can highly fluctuate in the long run without using the appropriate learning rate scheduling (Desjardins et al., 2010; 2009).”
An Efficient Learning Procedure for Deep Boltzmann Machines (2012, background): “Recently Tieleman (2008), Salakhutdinov and Hinton (2009a), Salakhut-dinov (2009), and Desjardins, Courville, Bengio, Vincent, and Delalleau (2010) have shown that this stochastic approximation algorithm, also termed persistent contrastive divergence, performs well compared to contrastive…”
Quickly Generating Representative Samples from an RBM-Derived Process (2011, methodology): “To estimate the quality of the samples generated by a sampling procedure, we use the indirect sampling likelihood (ISL) procedure (Breuleux, Bengio, & Vincent, 2010; Desjardins et al., 2010).”
Training restricted Boltzmann machines: An introduction (2014, background): “This can prevent the likelihood from diverging [5].”