What regularized auto-encoders learn from the data-generating distribution

By Guillaume Alain, Yoshua Bengio. 2012

551 citations recorded by the source corpus.

What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Generalized Denoising Auto-Encoders as Generative Models (2013, background): “The last step in this development (Alain and Bengio, 2013) generalized the result from Vincent (2011) by showing that when a denoising auto-encoder (or a contractive auto-encoder with the contraction on the whole encode/decode reconstruction function) is trained with small Gaussian corruption and…”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, neutral): “Larger temperatures on the right correspond to a ﬂatter distribution, so the whole destructive forward process maps the data distribution to a Gaussian and the creation process operates in reverse. strategy similar to that introduced in Alain and Bengio (2014).”
A Theory of Generative ConvNet (2016, methodology): “Also, our work gives an explicit hierarchical realization of auto-encoder based sampling (Alain & Bengio, 2014).”
GSNs : Generative Stochastic Networks (2015, background): “In a recent paper, Alain and Bengio (2013) showed that denoising auto-encoders with small Gaussian corruption and squared error loss estimated the score (derivative of the logdensity with respect to the input) of continuous observed random variables, thus implicitly estimating P (X).”
Representation Learning: A Review and New Perspectives (2012, background): “An interesting hypothesis emerges to answer that question, out of recent theoretical results (Vincent, 2011; Alain and Bengio, 2012): the training criterion of regularized auto-encoders, instead of being a form of maximum likelihood, corresponds to a different inductive principle, such as score…”
Synthesizing the preferred inputs for neurons in neural networks via deep generator networks (2016, background): “These generator networks are close to, but not true, generative models because they are trained without imposing any prior on the hidden distribution as in variational auto-encoders [14] or GANs [17], and without the addition of noise as in denoising auto-encoders [18].”
Paraphrasing Complex Network: Network Compression via Factor Transfer (2018, background): “One is an undercomplete convolutional autoencoder (CAE), the other is an overcomplete regularized autoencoder (RAE) which imposes l1 penalty on factors to learn the size of factors needed by itself [1].”
Towards Deep Neural Network Architectures Robust to Adversarial Examples (2014, methodology): “(2011b); Alain and Bengio (2012); Chen et al. (2014). We propose Deep Contractive Networks (DCNs), which incorporate a layer-wise contractive penalty, and show that adversarials generated from such networks have significantly higher distortion.”
Expert Gate: Lifelong Learning with a Network of Experts (2016, background): “It has been stated by [2] that in regularized (over-complete) autoencoders, the opposing forces between the risk and the regularization term result in a score like behavior for the reconstruction error.”
Deep Learning of Representations: Looking Forward (2013, methodology): “A Taylor expansion of the denoising error shows that it is also approximately equivalent to minimizing reconstruction error plus a contractive penalty on r(·) (Alain and Bengio, 2013).”
Deep Energy Estimator Networks (2018, background): “However, the “direct” score function estimation with MLPs are not robust — this problem was ﬁrst observed in Alain & Bengio (2014) for denoising autoencoders (see their Fig.”
Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser (2020, background): “The idea that denoising autoencoders learn the gradient of log density, and its connection to an underlying data manifold has been explored in [18, 39, 24].”
InfoVAE: Information Maximizing Variational Autoencoders (2017, methodology): “A similar model from the perspective of denoising autoencoders has been proposed in [3, 1] and from the perspective of modeling properties of human visual learning in [11].”
Denoising Criterion for Variational Auto-Encoding Framework (2015, background): “…performance (Vincent et al., 2008) because it makes the nearby data points in the low dimensional manifold to be robust against the presence of small noise in the high dimensional observation space (Seung, 1998; Vincent et al., 2008; Rifai, 2011; Alain & Bengio, 2014; Im et al., 2016).”
Saturating Auto-Encoders (2013, background): “From an energy based perspective, auto-encoders achieve low reconstruction cost in portions of the input space with high data density (recently, [8] has examined this perspective in depth).”
On Fenchel Mini-Max Learning (2019, methodology): “Our FML-based solution outperforms DAE score estimator [1] based DFM [64] in terms of FID, while giving similar performance in IS.”