Analyzing noise in autoencoders and deep networks

By Ben Poole, Jascha Narain Sohl-Dickstein, Surya Ganguli. 2014

115 citations recorded by the source corpus.

Autoencoders have emerged as a useful framework for unsupervised learning of internal representations, and a wide variety of apparently conceptually disparate regularization techniques have been proposed to generate useful features. Here we extend existing denoising autoencoders to additionally inject noise before the nonlinearity, and at the hidden unit activations. We show that a wide variety of previous methods, including denoising, contractive, and sparse autoencoders, as well as dropout can be interpreted using this framework. This noise injection framework reaps practical benefits by providing a unified strategy to develop new internal representations by designing the nature of the injected noise. We show that noisy autoencoders outperform denoising autoencoders at the very task of denoising, and are competitive with other single-layer techniques on MNIST, and CIFAR-10. We also show that types of noise other than dropout improve performance in a deep network through sparsifying, decorrelating, and spreading information across representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

A Unified Gradient Regularization Family for Adversarial Examples (2015, neutral): “85 % [17] Maxout + dropout + gradient p =∞ 0.”
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation (2019, background): “Prior work on efﬁcient coding [30] and autoencoders [38] has shown how ﬁlter properties change with noise in the unsupervised setting, resulting in lower-frequency ﬁlters with Gaussian , as we observe in Fig.”
Adversarial Dropout for Supervised and Semi-supervised Learning (2017, methodology): “Several techniques (Hinton et al. 2012; Poole, Sohl-Dickstein, and Ganguli 2014; Bishop 1995b; Lasserre, Bishop, and Minka 2006) have emerged over the past years to address this challenge, and dropout has become one of dominant methods due to its simplicity and effectiveness (Hinton et al. 2012;…”
The effectiveness of layer-by-layer training using the information bottleneck principle (2018, background): “Regularization by injection of noise is common practice in neural net training (Srivastava et al., 2014; Kingma et al., 2015; Wan et al., 2013; Poole et al., 2014), and was also analytically studied for the Gaussian IB case (Chechik et al.”
Learning Smooth Neural Functions via Lipschitz Regularization (2022, background): “Besides weight decay, there are other types of regularization that are not defined on network weights, such as adding noise [Poole et al. 2014] and Dropout [Srivastava et al. 2014].”
Towards Text Generation with Adversarially Learned Neural Outlines (2018, background): “This was shown to be equivalent to adding additive Gaussian noise to the hidden representations [41], where the variance of the noise is proportional to the contraction penalty in CAEs.”
A Survey on Deep Learning for Data-Driven Soft Sensors (2021, unknown): “Introducingnoiseintotheinput layercanalsoberegardedasakindofdataaugmentation[70], [71].”
Task-Oriented Feature Distillation (2020, methodology): “apply the orthogonal loss to the autoencoder to encourage the neural networks to learn orthogonal representations of the inputs [47, 34].”
R-Drop: Regularized Dropout for Neural Networks (2021, background): “To tackle with over-ﬁtting, many regularization techniques have been proposed, e.g., weight decay [33, 31, 28, 67], dropout [24, 64, 4, 66, 61], normalization [27, 57, 3, 26, 71], adding noise [25, 52], layer-wise pre-training and initialization [12, 21], label-smoothing [62], and so on.”
Neural Networks and Deep Learning (2018, unknown): “However, in this case, the noise has to be carefully calibrated [382].”