Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines

By Kyunghyun Cho, Tapani Raiko, Alexander Ilin. 2011

87 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “An alternative to centering is to push the mean activation toward zero by an appropriate activation function.”
Representation Learning: A Review and New Perspectives (2012, methodology): “Cho et al. (2011) proposes to use adaptive learning rates for RBM training, along with a novel and interesting idea for a gradient estimator that takes into account the invariance of the model to flipping hidden unit bits and inverting signs of corresponding weight vectors.”
Architecture Matters in Continual Learning (2022, methodology): “Skip connections (Cho et al., 2011), originally proposed for convolutional models by He et al. (2016), are crucial in the widely used ResNet architecture.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “To improve convergence and remove learning rates from the list of hyper-parameters, many authors have advocated exploring adaptive learning rate methods, either for a global learning rate (Cho et al., 2011), a layer-wise learning rate, a neuron-wise learning rate, or a parameter-wise learning rate (”
A Two-Stage Pretraining Algorithm for Deep Boltzmann Machines (2013, methodology): “5 It should be noted that these accuracies were computed purely to illustrate the effect of generative training of DBMs , and the reported accuracies are lower than other state-of-the-art accuracies (see, [15] for state-of-the-art accuracies for MNIST and [9] for Caltech-101 Silhouettes).”
Training Restricted Boltzmann Machines (2015, background): “First, the enhanced gradient was designed as an alternative update direction that can replace the gradient and leads to the desired invariance [4].”
Training restricted Boltzmann machines: An introduction (2014, background): “From this new formulation followed that the enhanced gradient (Cho et al., 2011) is equivalent to centering for a certain choice of offset parameters.”
Deep learning for visual understanding: A review (2016, background): “Further work in [56] discussed the main difficulties of training RBMs, their underlying reasons and proposed a new algorithm, which consists of an adaptive learning rate and an enhanced gradient, to address those difficulties.”
Tikhonov-Type Regularization for Restricted Boltzmann Machines (2012, methodology): “However, this approach makes it difﬁcult to utilize the enhanced gradient which has been shown to perform better than the traditional gradient [4].”
Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix (2015, methodology): “It has been observed that standard SGD training of RBMs is not invariant even to seemingly innocuous transformations of the data, such as inverting the pixel values (Cho et al., 2011).”
Deep learning for visual understanding (2017, unknown): “Further work in [82] discussed the main di culties of training RBMs, their underlying reasons and proposed a new algorithm, which consists of an adaptive learning rate and an enhanced gradient, to address those di culties.”
On Tracking The Partition Function (2011, methodology): “2 This same technique was recently used in [5], in the context of learning rate adaptation.”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, methodology): “Cho et al. (2011) proposes to use adaptive learning rates for RBM training, along with a novel and interesting idea for a gradient estimator that takes into account the invariance of the model to ﬂipping hidden unit bits and 13.”
Deep Boltzmann Machines and the Centering Trick (2012, background): “(2) Structural instability: As it has been identified by Cho et al. (2011), in standard Boltzmann machines, the weight matrix W tends to model in the first steps of the learning algorithm a global bias instead of co-dependencies between each units as we would expect.”
A Deep Neural Network for Acoustic-Articulatory Speech Inversion (2011, background): “However, other alternatives exist like persistent contrastive divergence [18], or enhanced gradient descent [19].”
Stochastic gradient estimate variance in contrastive divergence and persistent contrastive divergence (2013, methodology): “We use Adaptive learning rate (Cho et al., 2011) with an initial learning rate of 0.”