Parallel tempering is efficient for learning restricted Boltzmann machines

By Kyunghyun Cho, Tapani Raiko, Alexander Ilin. 2010

92 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Better Mixing via Deep Representations (2012, methodology): “To improve mixing, a powerful idea that has been explored recently for deep learning algorithms (Desjardins et al. , 2010; Cho et al. , 2010; Salakhutdinov, 2010b,a) is tempering (Neal, 1994).”
Neural Autoregressive Distribution Estimation (2016, methodology): “Another line of development has been to optimize likelihood using RobbinsMonro stochastic approximation (Younes, 1989), also known as Persistent CD (Tieleman, 2008), and develop good MCMC samplers for deep undirected models (Salakhutdinov, 2009, 2010; Desjardins et al., 2010; Cho et al., 2010).”
Representation Learning: A Review and New Perspectives (2012, background): “[44], and Salakhutdinov [173], [174] have all considered various forms of tempered transitions to address the failure of Gibbs chain mixing, and convincing solutions have not yet been clearly demonstrated.”
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (2019, methodology): “is Markov-chain Monte-Carlo (MCMC) sampling (Neal, 1993; Swendsen and Wang, 1986; Salakhutdinov, 2009; Desjardins et al., 2010; Cho et al., 2010).”
Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data (2016, methodology): “…update procedure proceeds in two phases: Dynamic generation of historical data (lines 14–22) As it has been shown in Desjardins et al (2010) and in Cho et al (2010) that RBMs can sample uniformly from the state space, we generate n ˆ B new training data points to represent past observations…”
Deep Learning of Representations: Looking Forward (2013, background): “Desjardins et al. (2010); Cho et al. (2010); Salakhutdinov (2010b,a) have all considered various forms of tempering to address the failure of Gibbs chain mixing in RBMs.”
Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines (2011, methodology): “Parallel tempering (PT) sampling was recently proposed to replace Gibbs sampling for estimating 〈·〉m (Desjardins et al., 2010; Cho et al., 2010).”
An Introduction to Restricted Boltzmann Machines (2012, methodology): “3 Parallel Tempering One of the most promising sampling technique used for RBM-training so far is parallel tempering (PT, [33,10,8]).”
Tikhonov-Type Regularization for Restricted Boltzmann Machines (2012, methodology): “We used parallel tempering to sample from the model distribution [7,3], and used the enhanced gradient and the adaptive learning rate, with both an initial learning rate and an upper-bound set to 0.”
Stochastic gradient estimate variance in contrastive divergence and persistent contrastive divergence (2013, methodology): “In addition, we intend to repeat the experiments with a larger data set, and train the models and replace the CD-1000 baseline with better approximations of exact sampling using e.g. Enhanced gradient (Cho et al., 2011) and parallel tempering (Cho et al., 2010).”
Training Restricted Boltzmann Machines (2015, methodology): “Parallel tempering was successfully applied for sampling in RBM training, where it was shown to lead to better generative models (Desjardins et al., 2010b; Cho et al., 2010).”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, background): “There have been a number of attempts made to address the failure of Gibbs chain mixing in the context of SML. Desjardins et al. (2010); Cho et al. (2010); Salakhutdinov (2010b,a) have all considered various forms of tempered transitions to improve the mixing rate of the negative phase Gibbs chain.”
A Two-Stage Pretraining Algorithm for Deep Boltzmann Machines (2013, methodology): “some work in applying more advanced sampling methods [32, 31, 14, 8].”
The flip-the-state transition operator for restricted Boltzmann machines (2013, methodology): “Restricted Boltzmann machines (RBMs, Smolensky 1986; Hinton 2002) are undirected graphical models describing stochastic neural networks.”
Training restricted Boltzmann machines: An introduction (2014, background): “3 Parallel tempering One of the most promising sampling techniques used for RBM training so far is parallel tempering (PT) [44, 12, 10].”