Deep Generative Stochastic Networks Trainable by Backprop

By Yoshua Bengio, Eric Thibodeau-Laufer, Guillaume Alain, Jason Yosinski. 2013

400 citations recorded by the source corpus.

We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised function approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood, along with a definition of an appropriate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows training to proceed with simple backprop, without the need for layerwise pretraining.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Nets (2014, methodology): “Adversarial models may also gain some statistical advantage from the generator network not being updated directly with data examples, but only with gradients ﬂowing through the discriminator.”
Generalized Denoising Auto-Encoders as Generative Models (2013, methodology): “l., 2000), in that the pairs (X t;X~ t 1) are not guaranteed to have the same asymptotic distribution as the pairs (X t;X~ t) as t!1. As a follow-up to the results in the next section, it is shown in Bengio et al. (2013a) that dependency networks can be cast into the same framework (which is that of”
MADE: Masked Autoencoder for Distribution Estimation (2015, background): “…algorithms for a variety of neural network models have been proposed (Bengio & Bengio, 2000; Larochelle & Murray, 2011; Gregor & LeCun, 2011; Uria et al., 2013; 2014; Kingma & Welling, 2014; Rezende et al., 2014; Bengio et al., 2014; Gregor et al., 2014; Goodfellow et al., 2014; Dinh et al., 2014).”
Unsupervised Visual Representation Learning by Context Prediction (2015, methodology): “For example, denoising autoencoders [46, 3] use reconstruction from noisy data as a pretext task; to tell the difference between noise and signal, the algorithm must connect images to other images with similar objects, in order to remove the parts of the image that don’t fit the shared appearance…”
Learning to generate chairs with convolutional neural networks (2014, methodology): “This includes a wide variety of methods ranging from Gaussian mixture models [9, 31] to autoregressive models [19] and stochastic variations of neural networks [2, 11, 25, 17, 30].”
Generalizing Hamiltonian Monte Carlo with Neural Networks (2017, background): “Recently, probabilistic models parameterized by deep neural networks have achieved great success at approximately sampling from highly complex, multi-modal empirical distributions [11, 6, 3, 20].”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, methodology): “Overall, our method of learning a transition operator outperforms previous attempts at learning transition operators (i.e. VAE, GSN and NET) using a local learning rule.”
A-NICE-MC: Adversarial Training for MCMC (2017, methodology): “This is drastically different from other iterative generative models trained with maximum likelihood, such as Generative Stochastic Networks (GSN, [18]) and Infusion Training (IF, [19]), because when we train T✓(xt+1|xt) we are not specifying a particular target for xt+1.”
Learning to Generate Samples from Noise through Infusion Training (2017, neutral): “Drawing on both Sohl-Dickstein et al. (2015) and the walkback procedure introduced for GSN in Alain et al. (2016), a variational variant of the walkback algorithm was investigated by Goyal et al. (2017) at the same time as our work.”
GSNs : Generative Stochastic Networks (2015, methodology): “After early versions of this work were published (Bengio et al., 2014), the GSN framework has been extended and applied to classification problems in several different ways (Goodfellow et al.”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, methodology): “Multiplying distributions is costly and difﬁcult for many techniques, including variational autoencoders, GSNs, NADEs, and most graphical models.”
Denoising Diffusion Implicit Models (2020, methodology): “Recent works on iterative generative models (Bengio et al., 2014), such as denoising diffusion probabilistic models (DDPM, Ho et al. (2020)) and noise conditional score networks (NCSN, Song & Ermon (2019)) have demonstrated the ability to produce samples comparable to that of GANs, without having…”
Generating Images with Perceptual Similarity Metrics based on Deep Networks (2016, background): “Recently, stochastic neural networks (Bengio et al., 2014; Kingma et al., 2014; Gregor et al., 2015) have become popular, and deterministic networks are being used for image generation tasks (Dosovit-skiy et al., 2015b).”
Conditional Generative Adversarial Nets (2014, neutral): “6 Deep GSN [2] 214 ± 1 .”
Generative Visual Manipulation on the Natural Image Manifold (2016, background): “, [19]) and their deep variants [20], auto-encoders [19,21] and more recently, stochastic neural networks [22,3,23] and deterministic networks [24].”
Generative adversarial networks (2014, neutral): “ractable for all but the most trivial instances, although they can be estimated by Markov chain Monte Carlo (MCMC) methods. Mixing poses a signiﬁcant problem for learning algorithms that rely on MCMC [3, 5]. Deep belief networks (DBNs) [16] are hybrid models containing a single undirected layer and ”
Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks (2015, methodology): “…parameterized, for example, by deep Boltzmann machines (e.g. Salakhutdinov & Hinton (2009), Goodfellow et al. (2013)) or by feed-forward neural networks (e.g. Bengio et al. (2014), Kingma et al. (2014)) –, or training autoencoder networks (e.g. Hinton & Salakhutdinov (2006), Vincent et al. (2008)).”
Adversarial Autoencoders (2015, methodology): “6 - 2110 ± 50 - Deep GSN [Bengio et al., 2014] 214 ± 1 .”
A note on the evaluation of generative models (2015, positive): “Following Bengio et al. (2013a), Parzen windows are in practice commonly fit to only 10,000 samples (e.g., Bengio et al., 2013b; Goodfellow et al., 2014; Li et al., 2015; Sohl-Dickstein et al., 2015).”