The "wake-sleep" algorithm for unsupervised neural networks.

By Geoffrey E. Hinton, Peter S. Dayan, Brendan J. Frey, Radford M. Neal. 1995

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Generative Adversarial Nets (2014, methodology): “This is similar to the inference net trained by the wake-sleep algorithm [15] but with the advantage that the inference net may be trained for a fixed generator net after the generator net has finished training.”
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, methodology): “In Appendix, we note how InfoGAN can be connected to the Wake-Sleep algorithm [27] to provide an alternative interpretation.”
Neural Variational Inference and Learning in Belief Networks (2014, methodology): “We also show that the resulting method trains sigmoid belief networks better than the wake-sleep algo-rithm (Hinton et al., 1995), which is the only algorithm we are aware of that is capable of training the same range of models efﬁciently.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “ reconstruct the input data was soon exploited to achieve representations that resemble simple cells in the brain [48]. The idea was later formalized in the context of probabilistic generative models [27], which have become a staple of modern unsupervised learning [28,22,35,56]. A key problem that g”
Unsupervised Visual Representation Learning by Context Prediction (2015, background): “To deal with these computational issues, a number of works, such as the wake-sleep algorithm [23], contrastive divergence [22], deep Boltzmann machines [45], and variational Bayesian methods [28, 43] use sampling to perform approximate inference.”
GSNs : Generative Stochastic Networks (2015, methodology): “This line of work started with the Helmholtz machine (Dayan et al., 1995; Hinton et al., 1995) and its wake–sleep algorithm.”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, methodology): “The wake-sleep algorithm [14, 6] introduced the idea of training inference and generative probabilistic models against each other.”
Generative adversarial networks (2014, neutral): “input to both Gand D. 2. Learned approximate inference can be performed by training an auxiliary network to predict z given x. This is similar to the inference net trained by the wake-sleep algorithm [15] but with the advantage that the inference net may be trained for a ﬁxed generator net after the”
Generative Moment Matching Networks (2015, methodology): “This is reminiscent of the recognition models used in the wake-sleep al-gorithm (Hinton et al., 1995), or variational auto-encoders (Kingma & Welling, 2014).”
Learning to See by Moving (2015, background): “Recent advances in computer vision have shown that visual features learnt by training neural networks for the task of object recognition using more than a million labelled images are useful for many computer vision tasks like semantic segmentation, object detection and action classification [22, 13,”
Toward Controlled Generation of Text (2017, methodology): “Wake-sleep al-gorithm (Hinton et al., 1995) introduced for learning deep directed graphical models shares similarity with VAEs by also combining an inference network with the generator.”
Unsupervised Learning of Visual Representations Using Videos (2015, background): “The first class of algorithms focus on learning generative models with strong priors [20, 46].”
On the challenges of learning with inference networks on sparse, high-dimensional data (2017, methodology): “More recently, Kingma & Welling (2014) and Rezende et al. (2014) proposed training a feedforward inference network (Hinton et al. , 1995) to ﬁnd good variational parameters ψ ( x ) for a given x , where ψ ( x ) is the output of a neural network with parameters φ that are trained to maximize L ( x ;…”
Tutorial on Variational Autoencoders (2016, background): “Another important way to look at the right hand side of Equation 5 is in terms of information theory, and in particular, the “minimum description length” principle which motivated many of the VAE’s predecessors like Helmholtz Machines [15], the Wake-Sleep Algorithm [16], Deep Belief Nets [17], and B”
Neural Variational Inference for Text Processing (2015, methodology): “…an inference network to approximate the variational distribution was ﬁrst proposed in the context of Helmholtz machines (Hinton & Zemel, 1994; Hinton et al., 1995; Dayan & Hinton, 1996), but applications of these directed generative models come up against the problem of establishing low…”