A Fast Learning Algorithm for Deep Belief Nets

By Geoffrey E. Hinton, Simon Osindero, Yee Whye Teh. 2006

16,970 citations recorded by the source corpus.

Subfields: Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Generative Adversarial Nets (2014, background): “Deep belief networks (DBNs) [16] are hybrid models containing a single undirected layer and several directed layers.”
Training Very Deep Networks (2015, background): “Finally, deep networks can be trained layer-wise to help in credit assignment [26, 27], but this approach is less attractive compared to direct training.”
FitNets: Hints for Thin Deep Nets (2014, background): “On the one hand, pre-training strategies, whether unsupervised (Hinton et al., 2006; Bengio et al., 2007) or supervised (Bengio et al., 2007) train the network parameters in a greedy lay-erwise fashion in order to initialize the network parameters in a potentially good basin of attraction.”
Deeply-Supervised Nets (2014, background): “However, even though hierarchical and recursive networks [7, 10, 12] have shown great promise in automatically learning thousands or even millions of features for pattern recognition, there nonetheless remain many fundamental open questions about deep learning.”
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “Early methods were based on stacked (often denoising) autoencoders or restricted Boltzmann machines [10–13].”
Neural Variational Inference and Learning in Belief Networks (2014, background): “Compared to powerful globally-normalized latent variable models, such as deep belief networks (Hinton et al., 2006) and deep Boltzmann machines (Salakhutdinov & Hinton, 2009a), which can now be trained on fairly large datasets, their purely directed counterparts have been left behind due to the…”
Generalized Denoising Auto-Encoders as Generative Models (2013, methodology): “The spirit of this procedure is thus very similar to the CD-k (Contrastive Divergence with k MCMC steps) procedure proposed to train RBMs (Hinton, 1999; Hinton et al. , 2006).”
Better Mixing via Deep Representations (2012, methodology): “The idea that deeper generative models produce not only better features for classification but also better quality samples (in the sense of better corresponding to the target distribution being learned) is not novel and several observations support this hypothesis already, some quantitatively (Salak”
A Generative Process for Contractive Auto-Encoders (2012, background): “Deep Learning algorithms learn a representation of input data x , which is typically used either to construct a classiﬁer 1 or in order to capture the structure of P ( x ).”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Generative approaches learn to generate or otherwise model pixels in the input space (Hinton et al., 2006; Kingma & Welling, 2013; Goodfellow et al., 2014).”
Music Transformer: Generating Music with Long-Term Structure (2018, background): “1 for more details) can be decomposed into a sequence of multi-hot pitch vectors, and their joint probability distributions can be captured using Restricted Boltzmann Machines (Smolensky, 1986; Hinton et al., 2006) or Neural Autoregressive Distribution Estimators (NADE; Larochelle & Murray, 2011).”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “Pretraining was once an essential technique for training deep networks (Hinton et al., 2006; Bengio et al., 2007; Dahl et al., 2010; Erhan et al., 2010).”
Neural Autoregressive Distribution Estimation (2016, background): “The most notable case is the Deep Belief Network (DBN; Hinton et al., 2006), which corresponds to a sigmoid belief network for which the prior over its top hidden layer is an RBM (whose hidden layer counts as an additional hidden layer).”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “ntations that resemble simple cells in the brain [48]. The idea was later formalized in the context of probabilistic generative models [27], which have become a staple of modern unsupervised learning [28,22,35,56]. A key problem that generative models must overcome, however, is that not all low-leve”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “One approach to semi-supervised learning involves unsupervised or self-supervised pretraining, followed by supervised fine-tuning [3, 4].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Learning good image representations is a key challenge in computer vision [1, 2, 3] as it allows for efficient training on downstream tasks [4, 5, 6, 7].”
What can the brain teach us about building artificial intelligence? (2016, background): “1995; Hinton et al. 1995) and variational optimization (Gregor et al. 2015; Mnih & Gregor 2014; Rezende et al. 2014), or nearest-neighbor density estimation (Kulkarni et al. 2015a; Stuhlmüller et al. 2013). One implication of amortization is that solutions to different problems will become correlate”