Saturating Auto-Encoders
By Rostislav Goroshin, Yann LeCun. 2013
43 citations recorded by the source corpus.
We introduce a simple new regularizer for auto-encoders whose hidden-unit activation functions contain at least one zero-gradient (saturated) region. This regularizer explicitly encourages activations in the saturated region(s) of the corresponding activation function. We call these Saturating Auto-Encoders (SATAE). We show that the saturation regularizer explicitly limits the SATAE's ability to reconstruct inputs which are not near the data manifold. Furthermore, we show that a wide variety of features can be learned when different activation functions are used. Finally, connections are established with the Contractive and Sparse Auto-Encoders.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Yoshua Bengio
- Sergey Levine
- Trevor Darrell
- Wanli Ouyang
- Andrew Zisserman
- Abhinav Kumar Gupta
- Andrea Vedaldi
- Jitendra Malik
Citations in the corpus
- Unsupervised Feature Learning from Temporal Data (2015, background): “A popular assumption in unsupervised learning is that high dimensional data lies on a low dimensional manifold parametrized by the latent variables [1, 18, 20, 6].”
- Unsupervised Learning of Spatiotemporally Coherent Metrics (2014, background): “A popular assumption in unsupervised learning is that high dimensional data lies on a low dimensional manifold parametrized by the latent variables as in [1, 19, 20, 7].”
- Biologically inspired protection of deep networks from adversarial attacks (2017, methodology): “Saturating autoencoders were introduced by Goroshin & LeCun (2013) as a means of explicitly encouraging activations to be in the saturating regime of the nonlinearity, in order to limit the autoencoder’s ability to reconstruct points that are not close by on the data manifold.”
- Multilingual Deep Learning (2013, background): “[7] proposed saturating autoencoder that explicitly limits the auto-encoder's ability to reconstruct the inputs which are not near the data manifold.”
- Deep learning for visual understanding (2017, unknown): “Saturating Autoencoder [108] Raises reconstruction error for inputs not near the data manifold Limits the ability to reconstruct inputs which are not near the data manifold Convolutional Autoencoder [109 111] Shares weights among all locations in the input, preserving spatial locality Utilizes the 2”