An Efficient Learning Procedure for Deep Boltzmann Machines

By Ruslan Salakhutdinov, Geoffrey E. Hinton. 2012

448 citations recorded by the source corpus.

We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent statistics are estimated using a variational approximation that tends to focus on a single mode, and data-independent statistics are estimated using persistent Markov chains. The use of two quite different techniques for estimating the two types of statistic that enter into the gradient of the log likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer pretraining phase that initializes the weights sensibly. The pretraining also allows the variational inference to be initialized sensibly with a single bottom-up pass. We present results on the MNIST and NORB data sets showing that deep Boltzmann machines learn very good generative models of handwritten digits and 3D objects. We also show that the features discovered by deep Boltzmann machines are a very effective way to initialize the hidden layers of feedforward neural nets, which are then discriminatively fine-tuned.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Learning Deep Structured Models (2014, background): “A multiplicity of variants have been proposed [10, 16, 8, 1, 22, 33] and shown to be extremely successful in a wide variety of applications including object detection, speech recognition as well as natural language processing.”
Knowledge Transfer in Deep Block-Modular Neural Networks (2015, background): “These successes can mainly be attributed to the use of pre-training [7,21], sharing [16], and various forms of regularization [8,10], as well as to increases in computational capacity and data availability [22].”
Sum-product networks: A new deep architecture (2011, background): “mating likelihood in DBNs or DBMs is a very challenging problem [30]; estimating marginals requires many Gibbs sampling steps that may take minutes or even hours, and the results are approximate without guarantee on the quality.”
Deep learning for computational biology (2016, neutral): “…for modelling images (Box 2 ), recurrent neural networks for sequential data (Sutskever, 2013 ; Lipton, 2015 ), or restricted Boltzmann machines (Salakhutdinov & Larochelle, 2010 ; Hinton, 2012 ) and autoencoders (Hinton & Salakhutdinov, 2006 ; Alain et al , 2012 ; Kingma & Welling, 2013 ) for…”
Noise as a Resource for Computation and Learning in Networks of Spiking Neurons (2014, background): “, the results on deep learning in hierarchical Boltzmann machines [29].”
Preliminary investigation of Boltzmann machine classifiers for speaker recognition (2012, methodology): “In practice, a mean field approximation is generally used, so that a vector of Bernoulli probabilities (that is, the probabilities Q(hj = 1|v)) rather than a binary vector is produced instead [2].”
On the Expressive Power of Deep Architectures (2011, background): “…and Haussler, 1994; Hinton et al., 2006; Tieleman, 2008; Salakhutdinov and Hinton, 2009; Desjardins et al., 2010) or Deep Boltzmann Machines (Salakhutdinov and Hinton, 2010; Lee et al., 2009), different flavours of auto-encoders (Bengio et al., 2007; Vincent et al., 2008; Larochelle et al.,…”
Deep learning for visual understanding: A review (2016, methodology): “To increase the ef ﬁ - ciency of DBMs, some researchers introduced an approximate inference algorithm [66,67], which utilizes a separate “ recognition ” model to initialize the values of the latent variables in all layers, thus effectively accelerating the inference.”
Deep Learning for Computer Vision: A Brief Review (2018, methodology): “These include accelerating inference by using separate models to initialize the values of the hidden units in all layers [47, 49], or other improvements at the pretraining stage [50, 51] or at the training stage [52, 53].”
Modeling of nonlinear system based on deep learning framework (2015, methodology): “A restricted Boltzmann machine [14] is an energybased model.”
A Deep Learning Scheme for Motor Imagery Classification based on Restricted Boltzmann Machines (2017, background): “The update of the weights in the other layers follows the backpropagation rules [33].”
Cambricon: An Instruction Set Architecture for Neural Networks (2016, background): “Among 10 representative benchmark networks listed in Table III, the DaDianNao ISA is only capable of expressing MLP, CNN, and RBM, but fails to implement the rest 7 benchmarks (RNN, LSTM, AutoEncoder, Sparse AutoEncoder, BM, SOM and HNN).”
A Two-Stage Pretraining Algorithm for Deep Boltzmann Machines (2013, background): “Recently, in [35] the same authors were able to show that the variational lower bound is guaranteed to increase by adding the top hidden layer using the proposed pretraining scheme.”
State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems (2017, background): “Very recently, however, there has been a breakthrough in the way deep learning systems such as Deep Belief Networks, Deep ANNs, and Deep Boltzmann Machines can lead to significant performance gain [16]–[19].”
Deep learning for visual understanding (2017, unknown): “To increase the e ciency of DBMs, some researchers introduced an approximate inference algorithm [92, 93], which utilizes a separate `recognition' model to initialize the values of the latent variables in all layers, thus e ectively accelerating the inference.”
Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis (2014, unknown): “…procedure after the initial bottom-up pass incorporates top-down feedback, which allows DBM to use higher-level knowledge to resolve uncertainty about intermediate-level features, thus creating better data-dependent representations and statistics for learning (Salakhutdinov and Hinton, 2012).”