Generalized Denoising Auto-Encoders as Generative Models

By Yoshua Bengio, L. Yao, Guillaume Alain, Pascal Vincent. 2013

597 citations recorded by the source corpus.

Recent work has shown how denoising and contractive autoencoders implicitly capture the structure of the data-generating density, in the case where the corruption noise is Gaussian, the reconstruction error is the squared error, and the data is continuous-valued. This has led to various proposals for sampling from this implicitly learned density function, using Langevin and Metropolis-Hastings MCMC. However, it remained unclear how to connect the training procedure of regularized auto-encoders to the implicit estimation of the underlying data-generating distribution when the data are discrete, or using other forms of corruption process and reconstruction errors. Another issue is the mathematical justification which is only valid in the limit of small corruption noise. We propose here a different attack on the problem, which deals with all these issues: arbitrary (but noisy enough) corruption, arbitrary reconstruction loss (seen as a log-likelihood), handling both discrete and continuous-valued variables, and removing the bias due to non-infinitesimal corruption noise (or non-infinitesimal contractive penalty).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Nets (2014, background): “Prominent recent work in this area includes the generative stochastic network (GSN) framework [5], which extends generalized denoising auto-encoders [4]: both can be seen as defining a parameterized Markov chain, i.”
Deep AutoRegressive Networks (2013, background): “Whilst it is possible to derive estimators that are unbiased (Bengio et al., 2013a), their empirical performance is often unsatisfactory.”
Deep Generative Stochastic Networks Trainable by Backprop (2013, methodology): “More recently, Bengio et al. (2013c) generalized this to arbitrary variables (discrete, continuous or both), arbitrary corruption (not necessarily asymptotically small), and arbitrary loss function (so long as they can be seen as a log-likelihood).”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “For example, generative stochastic networks (GSN [4, 1]) use denoising autoencoders to train a Markov chain whose equilibrium distribution matches the data distribution.”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, methodology): “VW is similar in spirit to DAE (Bengio et al., 2013b), and NET approaches (Sohl-Dickstein et al., 2015) but it retains two crucial differences.”
GSNs : Generative Stochastic Networks (2015, methodology): “The following theorem is proved by Bengio et al. (2013c).”
Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], specifically denoising score matching [22], which makes our work related to recent generative models using denoising score matching- and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
Generative adversarial networks (2014, neutral): “can be designed to be trained by back-propagation. Prominent recent work in this area includes the generative stochastic network (GSN) framework [5], which extends generalized denoising auto-encoders [4]: both can be seen as deﬁning a parameterized Markov chain, i.e., one learns the parameters of a ”
Generative Moment Matching Networks (2015, methodology): “This concept is referred to in the literature as disen-tangling factors of variation (Bengio et al., 2013a).”
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient (2016, background): “(Bengio et al. 2013) proposed denoising autoencoder (DAE) that learns the data distribution in a supervised learning fashion.”
Semi-supervised Learning with Ladder Networks (2015, background): “On the other hand, given a denoising function, one can draw samples from the corresponding distribution by creating a Markov chain that alternates between corruption and denoising [9].”
Collaborative Deep Learning for Recommender Systems (2014, background): “If we assume that both the clean input Xc and the corrupted input X0 are observed, similar to [4, 19, 3, 7], we can define the following generative process:”
Deep Supervised and Convolutional Generative Stochastic Network for Protein Secondary Structure Prediction (2014, methodology): “Models with deep representation have been exceptionally successful in capturing complex dependency in these data (Bengio et al., 2013a; Salakhutdinov & Hinton, 2009).”
What regularized auto-encoders learn from the data-generating distribution (2012, background): “…deep learning algorithms (Hinton et al., 2006; Bengio, 2009; Lee et al., 2009; Salakhutdinov and Hinton, 2009; Bengio and Delalleau, 2011; Bengio et al., 2013b) can perform better than their shallow counterparts, both in terms of learning features for the purpose of classiﬁcation tasks…”
Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models (2014, methodology): “Bengio et al. (2013) show how these models can be used as generative models, but since the generative process underlying the denoising auto-encoder is unknown, simulation from the model requires a slow Markov chain sampling procedure.”
LOGAN: Membership Inference Attacks Against Generative Models (2017, background): “of a given training dataset. In particular, generative models offer a way to artiﬁcially generate plausible images and videos and they are used in many applications, e.g., compression [60], denoising [7], inpainting [69], superresolution [35], semi-supervised learning [54], etc. In this paper, we st”
Deep Learning of Representations: Looking Forward (2013, background): “f multiple levels of representation. They have had important empirical successes in a number of traditional AI applications such as computer vision and natural language processing. See (Bengio, 2009; Bengio et al., 2013d) for reviews and Bengio (2013c) and the other chapters of the book by Montavon ”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, neutral): “In order to address the difficulty of sampling from pθ(x), we consider the recovery likelihood similar to Bengio et al. (2013), defined by the density of the sample conditioned on its noisy version perturbed by Gaussian noise.”
Learning Generative Models with Visual Attention (2013, background): “More recently, denoising auto-encoders have been proposed as a way to model the transition operator that has the same invariant distribution as the data generating distribution [5].”