Enhanced Gradient for Training Restricted Boltzmann Machines

By Kyunghyun Cho, Tapani Raiko, Alexander Ilin. 2013

67 citations recorded by the source corpus.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Generalized Denoising Auto-Encoders as Generative Models (2013, methodology): “h reported here. Optimization hyper-parameters (learning rate, momentum, and learning rate reduction schedule) were selected based on the training objective. We compare against a state-ofthe-art RBM (Cho et al., 2013) with an AIS log-likelihood estimate of -64.1 (AIS estimates tend to be optimistic)”
Neural Autoregressive Distribution Estimation (2016, background): “Work has also been directed towards proposing improved update rules or parameterization of the model’s energy function (Tieleman and Hinton, 2009; Cho et al., 2013; Montavon and M¨uller, 2012) as well as improved approximate inference of the hidden layers (Salakhutdinov and Larochelle, 2010).”
GSNs : Generative Stochastic Networks (2015, methodology): “However, the RBM samples can be improved by a spatial blur.”
Deep learning in neural networks: An overview (2014, background): “15) can be improved through gradient enhancements and automatic learning rate adjustments during stochastic gradient descent (Cho et al., 2013; Cho, 2014), and through Tikhonovtype (Tikhonov et al.”
Discrete Variational Autoencoders (2016, background): “…with a deep sigmoid belief network (RWS SBN; Bornschein & Bengio, 2015); the restricted Boltzmann machine (RBM; Smolensky, 1986), reporting the results of Cho et al. (2013); and the neural adaptive importance sampler with neural autoregressive distribution estimator (NAIS NADE; Du et al., 2015).”
Reweighted Wake-Sleep (2014, methodology): “, 2014) [4] (Salakhutdinov and Murray, 2008), [5] (Murray and Salakhutdinov, 2009), [6] (Cho et al., 2013).”
Iterative Neural Autoregressive Distribution Estimator NADE-k (2014, methodology): “Futhermore, NADE-5 has outperformed the previous best result obtained with the RBMs in (Cho et al., 2013), achieving the state-of-art result on this dataset.”
A Two-Stage Pretraining Algorithm for Deep Boltzmann Machines (2013, methodology): “In [27] the authors proposed to center the activations of the neurons in a DBM, which is closely related to the previously proposed method of the enhanced gradient for RBMs [9, 10].”
Wasserstein Training of Restricted Boltzmann Machines (2016, methodology): “We use the centered parameterization of the RBM for gradient descent [13] [3].”
Deep Learning for Computer Vision: A Brief Review (2018, background): “A detailed explanation along with the description of a practical way to train RBMs was given in [37], whereas [38] discusses the main difficulties of training RBMs and their underlying reasons and proposes a new algorithm with an adaptive learning rate and an enhanced gradient, so as to address the ”
Bounding the Test Log-Likelihood of Generative Models (2013, methodology): “y cost, trained for 300 epochs, best model (at epoch 250) selected by visual inspection of generated samples. DBN-2:architecture is 784-4000-1000, trained layerwise, the ﬁrst layer being the RBM from Cho et al. (2013) that has a near state-of-the-art AIS -64.1, second layer trained by PCD-9, the ent”
Learning Deep Sigmoid Belief Networks with Data Augmentation (2015, background): “Remarkably, our implementation of FVSBN beats the state-of-the-art results on this dataset (Cho et al., 2013) by 10 nats.”
Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix (2015, background): “Several researchers independently discovered an update rule for training RBMs and DBMs, known as the enhanced gradient (Cho et al., 2011; 2013), or centering trick (Montavon & Muller, 2012), which improves the stability of SGD and is invariant to inverting pixel values.”