GSNs : Generative Stochastic Networks

By Guillaume Alain, Yoshua Bengio, L. Yao, Jason Yosinski, Eric Thibodeau-Laufer, Saizheng Zhang, Pascal Vincent. 2015

50 citations recorded by the source corpus.

We introduce a novel training principle for generative probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework generalizes Denoising Auto-Encoders (DAE) and is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution is a conditional distribution that generally involves a small move, so it has fewer dominant modes and is unimodal in the limit of small moves. This simplies the learning problem, making it less like density estimation and more akin to supervised function approximation, with gradients that can be obtained by backprop. The theorems provided here provide a probabilistic interpretation for denoising autoencoders and generalize them; seen in the context of this framework, auto-encoders that learn with injected noise are a special case of GSNs and can be interpreted as generative models. The theorems also provide an interesting justication for dependency networks and generalized pseudolikelihood and dene an appropriate joint distribution and sampling mechanism, even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. Experiments validating these theoretical results are conducted on both synthetic datasets and image datasets. The experiments employ a particular architecture that mimics the Deep Boltzmann Machine Gibbs sampler but that allows training to proceed with backprop through a recurrent neural network with noise injected inside and without the need for layerwise pretraining.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, methodology): “Other methods for learning transition operators of Markov chains include infusion training [2], variational walkback [13], generative stochastic networks [1], and others [47, 51, 33, 39].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, neutral): “For example, generative stochastic networks (GSN [4, 1]) use denoising autoencoders to train a Markov chain whose equilibrium distribution matches the data distribution.”
Learning to Generate Samples from Noise through Infusion Training (2017, methodology): “…backpropagation • Generative uses of denoising autoencoders (Vincent et al., 2010) and their generalization as Generative Stochastic Networks (Alain et al., 2016) • Inverting a non-equilibrium thermodynamic slow diffusion process (Sohl-Dickstein et al., 2015) • Continuous transformation of a…”
Unrolled Generative Adversarial Networks (2016, background): “…2013; Rezende et al., 2014; Gregor et al., 2015; Kulkarni et al., 2015; Burda et al., 2015; Kingma et al., 2016), generative stochastic networks (Alain et al., 2015), diffusion probabilistic models (Sohl-Dickstein et al., 2015), autoregressive models (Theis & Bethge, 2015; van den Oord et al.,…”
Dataset Augmentation in Feature Space (2017, background): “…to interest in such techniques, leading to, e.g., variational autoencoders (Kingma & Welling, 2014), generative adversarial networks (Goodfellow et al., 2014), and generative stochastic networks (Alain et al., 2016), each of which could be used to generate useful feature spaces for augmentation.”
Deep Variational Canonical Correlation Analysis (2016, background): “Recently, there has been much interest in unsupervised deep generative models (Kingma & Welling, 2014; Rezende et al., 2014; Goodfellow et al., 2014; Gregor et al., 2015; Makhzani et al., 2016; Burda et al., 2016; Alain et al., 2016).”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “Closely related are diffusion models [1], [11], [86], [197] which gradually destroy data x0 by adding noise over a fixed number of steps T using a noise schedule b1:T determined so that xT is approximately normally distributed.”
Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains (2018, methodology): “Neural network models are another option, such as Generative Adversarial Networks [Goodfellow et al., 2014] and Stochastic Neural Networks [Sohn et al., 2015; Alain et al., 2016].”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “Generative modeling by learning stochastic processes has a long history (Movellan, 2008; Lyu, 2009; Sohl-Dickstein et al., 2011; Bengio et al., 2014; Alain et al., 2016; Goyal et al., 2017; Bordes et al., 2017; Song & Ermon, 2019; Ho et al., 2020).”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Furthermore, diffusion models can be combined with more other powerful generative models, such as Generative Stochastic Networks [2] and Generative Flow Networks [16].”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, background): “Diffusion-based models (Sohl-Dickstein et al., 2015; Ho et al., 2020) learn the finite-time reversal of a diffusion process, sharing the idea of learning transition operators of Markov chains with Goyal et al. (2017); Alain et al. (2016); Bordes et al. (2017).”