Scheduled denoising autoencoders

By Krzysztof J. Geras, Charles Sutton. 2014

49 citations recorded by the source corpus.

We present a representation learning method that learns features at multiple different levels of scale. Working within the unsupervised framework of denoising autoencoders, we observe that when the input is heavily corrupted during training, the network tends to learn coarse-grained features, whereas when the input is only slightly corrupted, the network tends to learn fine-grained features. This motivates the scheduled denoising autoencoder, which starts with a high level of noise that lowers as training progresses. We find that the resulting representation yields a significant boost on a later supervised task compared to the original input, or to a standard denoising autoencoder trained at a single noise level. After supervised fine-tuning our best model achieves the lowest ever reported error on the CIFAR-10 data set among permutation-invariant methods.

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Generative Modeling by Estimating Gradients of the Data Distribution (2019, background): “In [14, 7, 66], different annealing schedules on the noise for training denoising autoencoders are proposed.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], specifically denoising score matching [22], which makes our work related to recent generative models using denoising score matching- and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
VConv-DAE: Deep Volumetric Shape Learning Without Object Labels (2016, background): “Recent work of Geras and Sutton [13] experiment with diﬀerent noise levels used in DAE, starting with high levels and then reducing to low noise levels.”
Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching (2019, methodology): “Our approach is conceptually similar to the idea of combining denoising autoencoder and annealing (Geras and Sutton, 2015; Chandra and Sharma, 2014; Zhang and Zhang, 2018) though this idea was proposed in the context of pre-training neural networks for classification applications.”
Do Deep Convolutional Nets Really Need to be Deep and Convolutional? (2016, result): “n accuracy of 70.2%. We believe this to be the most accurate shallow MLP ever reported for CIFAR-10 (in comparison to 63.1% achieved by Le et al. (2013), 63.9% by Memisevic et al. (2015) and 64.3% by Geras and Sutton (2015)). Although this model cannot compete with convolutional models, clearly dist”
Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces (2019, methodology): “Our approach is conceptually similar to the idea of combining denoising autoencoder and annealing (Geras and Sutton, 2015; Chandra and Sharma, 2014; Zhang and Zhang, 2018) though they were proposed in the context of pre-training neural networks for classification applications.”
Convolutional adaptive denoising autoencoders for hierarchical feature extraction (2018, methodology): “The performance of our algorithm is higher than those of a few of these methods (such as Convolutional net LeNet-5 [24], Conv.”