Extracting and composing robust features with denoising autoencoders

By Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol. 2008

8,231 citations recorded by the source corpus.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Generative Adversarial Nets (2014, methodology): “Some models such as denoising auto-encoders [30] and contractive autoencoders have learning rules very similar to score matching applied to RBMs.”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “SDAE is also consistently better than SAE, which aligns with other findings that adding noise to AEs produces richer representations (Vincent et al., 2008).”
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, background): “Another intriguing line of work consists of the ladder network [14], which has achieved spectacular results on a semi-supervised variant of the MNIST dataset.”
Deep AutoRegressive Networks (2013, background): “Recently several authors (Ranzato et al., 2007; Vincent et al., 2008; Vincent, 2011; Rifai et al., 2012; Bengio et al., 2013b) have developed probabilistic versions of regularised au-toencoders, along with means of generating samples from such models.”
Pylearn2: a machine learning research library (2013, methodology): “Pylearn2 has implementations of several models that were developed at LISA, including denoising auto-encoders (DAEs) [53], contractive auto-encoders (CAEs) [42] including higher-order CAEs [43], spike-and-slab RBMs (ssRBMs) [11] including ssRBMs with pooling [12], reconstruction sampling…”
Deep Generative Stochastic Networks Trainable by Backprop (2013, background): “, 2006), auto-encoder variants (Bengio et al., 2007; Vincent et al., 2008), and sparse coding variants (Lee et al.”
Masked Autoencoders Are Scalable Vision Learners (2021, background): “The idea of masked autoencoders, a form of more general denoising autoencoders [48], is natural and applicable in computer vision as well.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, neutral): “, denoising auto-encoders [58], context autoencoders [48], or cross-channel auto-encoders (colorization) [64, 65].”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “Our approach is based on training a denoising auto-encoder (Vincent et al., 2008) that encodes sentences and paragraphs in a shared space.”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “The second baseline is DAE , which simply uses denoising auto-encoder (Vincent et al., 2008) to pre-train the encoder and decoder.”
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, background): “Dropout noise (Hinton et al. , 2012) and masking noise (in denoising auto-encoders (Vincent et al. , 2008)) is multiplied (just after the neuron non-linearity), while in semantic hashing (Salakhutdinov and Hinton, 2009) noise is added (just before the non-linearity).”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, methodology): “The idea of noise injection has previously been used in the context of Denoising Auto-Encoders (DAEs) (Vincent et al., 2008), where noise is added to the inputs and the model is trained to re-construct the original inputs.”
Neural Autoregressive Distribution Estimation (2016, methodology): “The resulting training procedure resembles that of a denoising autoencoder (Vincent et al., 2008).”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Denoising autoencoding (Vincent et al., 2008) trains a model that recovers original data from randomly corrupted versions, and is widely used as an effective self-supervised representation learning and pretraining strategy, e.g., in computer vision (Erhan et al., 2010) and natural language…”
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (2020, background): “% 67.6 70.7 86.1 % 47.4 47.4 49.1 The idea of using unlabeled data to networks has been shown effective across many ﬁelds, including computer vision (Vincent et al., 2008; Erhan et al., 2010), NLP (Peters et al., 2018; Devlin et al., 2019), graphs (Hu et al., 2020), and programming languages (Feng…”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Many of these approaches rely either on auto-encoding of images [24, 25, 26] or on adversarial learning [27], jointly modelling data and representation [28, 29, 30, 31].”
Visualizing and Understanding Convolutional Networks (2013, background): “As such, this is a form of supervised pre-training, which contrasts with the unsupervised pre-training methods popularized by (Hinton et al., 2006) and others (Bengio et al., 2007; Vincent et al., 2008).”
Challenges in representation learning: A report on three machine learning contests (2013, background): “Other top scorers included Jingjing Xie, Bing Xu and Zhang Chuang, that developed ensemble voting techniques for use with denoising autoencoders [10] and maxout networks [11].”