A Generative Process for Contractive Auto-Encoders

By Salah Rifai, Yoshua Bengio, Yann Dauphin, Pascal Vincent. 2012

51 citations recorded by the source corpus.

The contractive auto-encoder learns a representation of the input data that captures the local manifold structure around each data point, through the leading singular vectors of the Jacobian of the transformation from input to representation. The corresponding singular values specify how much local variation is plausible in directions associated with the corresponding singular vectors, while remaining in a high-density region of the input space. This paper proposes a procedure for generating samples that are consistent with the local structure captured by a contractive auto-encoder. The associated stochastic process defines a distribution from which one can sample, and which experimentally appears to converge quickly and mix well between modes, compared to Restricted Boltzmann Machines and Deep Belief Networks. The intuitions behind this procedure can also be used to train the second layer of contraction that pools lower-level features and learns to be invariant to the local directions of variation discovered in the first layer. We show that this can help learn and represent invariances present in the data and improve classification error.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Generative Adversarial Nets (2014, background): “[7] and used for various generative models for which the exact likelihood is not tractable [24, 3, 4].”
Deep AutoRegressive Networks (2013, background): “Recently several authors (Ranzato et al., 2007; Vincent et al., 2008; Vincent, 2011; Rifai et al., 2012; Bengio et al., 2013b) have developed probabilistic versions of regularised au-toencoders, along with means of generating samples from such models.”
Deep Generative Stochastic Networks Trainable by Backprop (2013, methodology): “ls trained on MNIST. The LL is not directly comparable to AIS likelihood estimates because we use a Gaussian mixture rather than a Bernoulli mixture to compute the likelihood, but we can compare with Rifai et al. (2012); Bengio et al. (2013b;c) (from which we took the last three columns). A DBN-2 ha”
Generalized Denoising Auto-Encoders as Generative Models (2013, methodology): “Another breakthrough has been the development of an empirically successful sampling algorithm for contractive auto-encoders (Rifai et al. , 2012), which basically involves composing encoding, decoding, and noise addition steps.”
Better Mixing via Deep Representations (2012, methodology): “The CAEs have sizes 768-1000-1000 and 2304-1000-1000 on MNIST and TFD respectively.”
GSNs : Generative Stochastic Networks (2015, methodology): “ls trained on MNIST. The LL is not directly comparable to AIS likelihood estimates because we use a Gaussian mixture rather than a Bernoulli mixture to compute the likelihood, but we can compare with Rifai et al. (2012); Bengio et al. (2013b,c) (from which we took the last three columns). A DBN-2 ha”
Generative adversarial networks (2014, neutral): “ans was obtained by cross validation on the validation set. This procedure was introduced in Breuleux et al. [8] and used for various generative models for which the exact likelihood is not tractable [25, 3, 5]. Results are reported in Table 1. This method of estimating the likelihood has somewhat h”
Generative Moment Matching Networks (2015, background): “More related to our own work, there is a line of research devoted to recovering density models from auto-encoder networks using MCMC procedures (Rifai et al., 2012; Bengio et al., 2013b; 2014).”
Representation Learning: A Review and New Perspectives (2012, methodology): “On the other hand, denoising reconstruction error is clearly immune to this problem, so that solves the problem for DAEs.”
A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, background): “The contractive AE can be sampled [69], that is, it can generate new instances from the learned model, by using the Jacobian of the encoder to add a small noise to another point and computing its codiﬁcation.”
Deep Learning of Representations: Looking Forward (2013, methodology): “This can be used to endow regularized auto-encoders with a probabilistic interpretation and to sample from the implicitly learned density models (Rifai et al., 2012b; Bengio et al., 2012; Alain and Bengio, 2013) through some variant of Langevin or Metropolis-Hastings Monte-Carlo Markov chains…”
Knowledge Matters: Importance of Prior Information for Optimization (2013, methodology): “CAE+MLP with Supervised Finetuning: A regularized auto-encoder which sometimes outperforms the DAE is the Contractive Auto-Encoder (CAE), (Rifai et al., 2012), which penalizes the Frobenius norm of the Jacobian matrix of derivatives of the hidden units with respect to the CAE’s inputs.”
High-Dimensional Probability Estimation with Deep Density Models (2013, background): “Unfortunately, the autoencoder does not have a clear probabilistic interpretation (although see [Rifai et al., 2012] for a discussion).”
Bounding the Test Log-Likelihood of Generative Models (2013, methodology): “per-parameter ˙must be tuned according to, for instance, the log-likelihood of a validation set. This estimator was recently used to assess the qualities of the generative models such as stacked CAE (Rifai et al., 2012b), restricted Boltzmann machines (RBM, Desjardins et al., 2010), deep belief netw”
From Variational to Deterministic Autoencoders (2019, background): “Heuristic attempts to equip them with a generative mechanism include MCMC schemes [6,40].”
Tackling Over-pruning in Variational Autoencoders (2017, methodology): “We measure sample quality using the Parzen window estimator (Rifai et al., 2012).”
Representation Learning and Adversarial Generation of 3D Point Clouds (2017, background): “• Older work on sampling from autoencoders [RBDV12, BYAV13, BLAY14].”