The Helmholtz Machine

By Peter S. Dayan, Geoffrey E. Hinton, Radford M. Neal, Richard S. Zemel. 1995

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, background): “A popular framework for unsupervised learning is that of representation learning [1, 2], whose goal is to use unlabelled data to learn a representation that exposes important semantic features as easily decodable factors.”
Neural Variational Inference and Learning in Belief Networks (2014, methodology): “This algorithm was introduced for training Helmholtz machines (Dayan et al., 1995), which are multi-layer belief networks augmented with recognition networks.”
Neural Autoregressive Distribution Estimation (2016, methodology): “Work on directed generative networks dates back to the original work on sigmoid belief networks (Neal, 1992) and the Helmholtz machine (Hinton et al., 1995; Dayan et al., 1995).”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “Energy-based models (EBMs) have a long history in statistics and machine learning [16, 70, 43].”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, background): “Other than exact likelihood models, there are also a lot of works that model natural images with a Helmholtz Machine (Dayan et al., 1995) or variants thereof (Kingma & Welling, 2013; Rezende et al.”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, methodology): “…of two classes: (1) directed graphical models, corresponding to a ﬁnite time feedforward generative process (e.g. variants of the Helmholtz machine (Dayan et al., 1995) like the Variational Auto-Encoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014)), or (2) energy function based…”
Density estimation using Real NVP (2016, background): “Recent advances in stochastic variational inference [27] and amortized inference [13, 43, 35, 49], allowed efficient approximate inference and learning of deep directed graphical models by maximizing a variational lower bound on the log-likelihood [45].”
GSNs : Generative Stochastic Networks (2015, methodology): “This line of work started with the Helmholtz machine (Dayan et al., 1995; Hinton et al., 1995) and its wake–sleep algorithm.”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, methodology): “The wake-sleep algorithm [14, 6] introduced the idea of training inference and generative probabilistic models against each other.”
NICE: Non-linear Independent Components Estimation (2014, methodology): “s (Hyv¨arinen and Pajunen, 1999) via ensemble learning (Roberts and Everson, 2001; Lappalainen et al., 2000) and in (Kingma and Welling, 2014; Rezende et al., 2014) using a type of Helmholtz machine (Dayan et al., 1995). Generative adversarial networks (GAN) (Goodfellow et al., 2014) also train a ge”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “Energy-based models are another class of likelihood-based models with a rich history [1, 13, 30].”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, methodology): “We adopt this base architecture as it is simple, empirically effective, and has been postulated to resemble biological processes of perception (Dayan et al., 1995).”
Unrolled Generative Adversarial Networks (2016, background): “…et al., 2015), autoregressive models (Theis & Bethge, 2015; van den Oord et al., 2016a;b), real non-volume preserving transformations (Dinh et al., 2014; 2016), Helmholtz machines (Dayan et al., 1995; Bornschein et al., 2015), and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014).”
DRAW: A Recurrent Neural Network For Image Generation (2015, background): “In the context of generative neural networks, this typically means that all the pixels are conditioned on a single latent distribution (Dayan et al., 1995; Hinton & Salakhut-dinov, 2006; Larochelle & Murray, 2011).”
Importance Weighted Autoencoders (2015, methodology): “A classic approach was the wake-sleep algorithm, used to train Helmholtz machines (Dayan et al., 1995).”
Sparse coding with an overcomplete basis set: a strategy employed by V1? (1997, background): “Surely, there will be statistical dependencies among the elements of the single-stage model, and it would be desirable to have these modeled by a second and third stage in a hierarchical fashion (Dayan et al., 1995; Lewicki & Sejnowski, 1996).”
Ladder Variational Autoencoders (2016, background): “A line of motivation, already noted in [3], is that a purely bottom-up inference process as in i.e. VAEs does not correspond well with real perception, where iterative interaction between bottom-up and top-down signals produces the ﬁnal activity of a unit 4 .”
DeepCoder: Learning to Write Programs (2016, neutral): “See Appendix E for more details.”