Deep belief networks

By Geoffrey E. Hinton. 2009

540 citations recorded by the source corpus.

Recently, Hinton et al.[6] derived a way to perform fast, greedy learning of deep belief networks (DBN) one layer at a time, with the top two layers forming an undirected bipartite graph (associate memory). The learning procedure consists of training a stack of Restricted Boltzmann Machines (RBM’s) each having only one layer of latent (hidden) feature detectors. The learned feature activations of one RBM are used as the “data” for training the next RBM in the stack. The important aspect of this layer-wise training procedure is that, provided the number of features per layer does not decrease, [6] showed that each extra layer increases a variational lower bound on the log probability of data. So layer-by-layer training can be repeated several times 1 to learn a deep, hierarchical model in which each layer of features captures strong high-order correlations between the activities of features in the layer below. We will discuss three ideas based on greedily learning a hierarchy of features: Nonlinear Dimensionality Reduction. The DBN framework allows us to make nonlinear autoencoders work considerably better [7] than widely used methods such as PCA, SVD, and LLE. The standard way to train autoencoders is to use backpropagation to reduce the reconstruction error. It is difﬁcult, however, to optimize the weights in non-linear autoencoders that have multiple hidden layers with many million parameters [3, 5]. We use our greedy learning algorithm to pretrain autoencoders. This pretraining stage discovers useful features efﬁciently. After the pretraining stage, the model is “unfolded” to produce encoder and decoder networks that initially use the same weights. The global ﬁne-tuning stage then uses backpropagation through the whole autoencoder to ﬁne-tune the weights for optimal reconstruction. The key idea is that the greedy learning algorithm

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “These include DBNs ( Deep Belief Networks ) [10], CNNs ( Convolutional Neural Networks ) [11], RNNs ( Recurrent Neural Networks ) [12] as well as LSTMs ( Long Short-Term Memory ) [13] or AEs ( autoencoders ).”
AI-Based Modeling: Techniques, Applications and Research Issues Towards Automation, Intelligent and Smart Systems (2022, background): “…Generative Adversarial Network (GAN) [32], Autoencoder (AE) [31], Restricted Boltzmann Machine (RBM) [58], Self-Organizing Map (SOM) [50], and Deep Belief Network (DBN) [39], as well as their variants, can be used to build the deep generative learning models to solve the relevant real-world issues.”
A Review on a Deep Learning Perspective in Brain Cancer Classification (2019, background): “The most common DL models for brain image characterization are convolution neural networks (CNN) [85], auto encoders [86] and deep belief networks (DBNs) [87].”
Understanding of Machine Learning with Deep Learning: Architectures, Workflow, Applications and Future Directions (2023, background): “In a paper they released in 2006, the authors of [50] introduced DBN. DBN is a collection of restricted Boltzmann machines (RBMs).”
MolTrans: Molecular Interaction Transformer for drug–target interaction prediction (2020, methodology): “They often take drug and protein data as inputs, cast DTI as a classification problem, and make prediction by feeding the inputs through deep learning models such as deep neural network (DNN) (Unterthiner et al., 2014), deep belief network (DBN) (Wen et al., 2017) and convolutional neural network (C”
Breast cancer classification using deep belief networks (2016, background): “On the other and, Deep Belief Network (DBN) is a generative graphical model, or lternatively a type of deep neural network, composed of multiple ayers of latent variables (“hidden units”), with connections between he layers but not between units within each layer (Hinton, 2009b).”
The present and future of deep learning in radiology. (2019, background): “Deep Belief Network (DBN) [62] is another example unsupervised deep neural network which is trained layer-by-layer to reduce reconstruction error.”
Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study (2020, background): “Refer to [149], the probability of generating a visible vector, V , can be calculated as: where Prob ( H | W ) is the prior distribution over hidden vectors.”
A Survey on Deep Learning (2018, background): “DBN [61] is a hybrid probabilistic generative model in which a typical RBM with undirected connections is formed by the top two layers, and the lower layers use directed connections to receive inputs from the layer above.”
Cancer Diagnosis Using Deep Learning: A Bibliographic Review (2019, methodology): “Using the stack of Restricted Boltzmann Machines (RBM) [134], a probabilistic generative model is constructed, which is named as Deep belief network (DBN) given in Figure 6.”
Feature dimensionality reduction: a review (2022, methodology): “A deep belief network (DBN) is also a common unsupervised deep learning method to reduce the dimension of the feature vectors and extract features from high-dimensional data [159].”