Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

By Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol. 2010

7,579 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Distributed Representations of Sentences from Unlabelled Data (2016, methodology): “Transforming data into DAE representations (as a ‘pre-training’ or initialisation step) gives more robust (supervised) classification performance in deep feedforward networks (Vincent et al., 2010).”
Auto-Encoding Variational Bayes (2013, positive): “In relevant recent work on autoencoders [VLL + 10] it was shown that the training criterion of un-regularized autoencoders corresponds to maximization of a lower bound (see the infomax principle [Lin89]) of the mutual information between input X and latent representation Z .”
Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “The pioneering work of [59] presents masking as a noise type in DAE.”
Neural Discrete Representation Learning (2017, background): “Learning representations with continuous features have been the focus of many previous work [16, 39, 6, 9] however we concentrate on discrete representations [27, 33, 8, 28] which are potentially a more natural ﬁt for many of the modalities we are interested in. Language is inherently discrete,…”
Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns (2019, methodology): “We want to emphasize that (1) our models differ from denoising auto-encoders [73], where the training data are fully observed, and noises are added as a matter of regularization; (2) our experiments are different from in painting or de-noising, where the prior model or regularization has already bee”
Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, neutral): “Such approach towards symmetry is consistent with both empirical observations (Vincent et al., 2010) and theoretical analysis (Arora et al., 2015) of auto-encoders, for which symmetric weights are associated with minimizing reconstruction error.”
Learning to Generate Samples from Noise through Infusion Training (2017, background): “Earlier works that propose to directly learn a transition operator resulted from research to turn au-toencoder variants that have a stochastic component, in particular denoising autoencoders (Vincent et al., 2010), into generative models that one can sample from.”
Towards Conceptual Compression (2016, methodology): “Numerous techniques exist for unsupervised learning in deep networks, e.g. sparse auto-encoders and sparse coding (Kavukcuoglu et al., 2010; Le, 2013), denoising autoencoders (Vincent et al., 2010), deconvolutional networks (Zeiler et al., 2010), restricted Boltzmann machines (Hinton &…”
Unrolled Generative Adversarial Networks (2016, background): “This success has been achieved through a surprising diversity of training losses and model architectures, including denoising autoencoders (Vincent et al., 2010), variational autoencoders (Kingma & Welling, 2013; Rezende et al.”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, background): “Filling-in-the-blanks tasks, proposed earlier in (Vincent et al., 2008; 2010), later proved remarkably successful in learning potent representations for natural language processing (Vaswani et al., 2017; Devlin et al., 2019).”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “Generative models are also commonly used for building representations (Vincent et al., 2010; Kingma et al., 2014; Salimans et al., 2016; Rezende et al., 2016; Donahue et al., 2016), and mutual information (MI) plays an important role in the quality of the representations they learn.”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, background): “The latter includes objectives such as generative autoencoder image reconstruction [46], triplets [44] and spatial-temporal order or context prediction [36, 12, 16], for example predicting patch proximity [29], solving jig-saw puzzles [40] and inpainting [41].”
Improved Regularization of Convolutional Neural Networks with Cutout (2017, background): “Denosing auto-encoders [18] and context encoders [14] both rely on self-supervised learning to elicit useful feature representations of images.”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “However, this objective function can lead to degenerate solutions because the questions can be thought of as an overcomplete representation of the answers (Vincent et al., 2010).”
Learning Robust and Multilingual Speech Representations (2020, neutral): “, 2012), as well as other application areas (Hinton et al., 2006; Bengio et al., 2007; Vincent et al., 2010).”
An Unsupervised Autoregressive Model for Speech Representation Learning (2019, methodology): “This makes the autoregressive loss more appealing, because no additional techniques, such as denoising [6], are required to avoid the trivial solution as for the autoencoding loss.”
Representation Learning: A Review and New Perspectives (2012, background): “7.2.3 Contractive Auto-Encoders Contractive Auto-Encoders (CAE), proposed by Rifai et al. (2011a), follow up on Denoising Auto-Encoders (DAE) and share a similar motivation of learning robust representations.”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Typically, a parametrized mapping is learned between a predefined random noise and the images, with either an autoencoder [4, 22, 29, 40, 62], a generative adversarial network (GAN) [20] or more directly with a reconstruction loss [6].”