Higher Order Contractive Auto-Encoder

By Salah Rifai, Grégoire Mesnil, Pascal Vincent, Xavier Muller, Yoshua Bengio, Yann Dauphin, Xavier Glorot. 2011

283 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Representation Learning: A Review and New Perspectives (2012, methodology): “7.2.3 Contractive Auto-Encoders Contractive Auto-Encoders (CAE), proposed by Rifai et al. (2011a), follow up on Denoising Auto-Encoders (DAE) and share a similar motivation of learning robust representations.”
Semi-supervised Learning with Ladder Networks (2015, methodology): “There is an interesting connection between our Γ-model and the contractive cost used by Rifai et al. (2011a): a linear denoising function ẑ i = aiz̃ (L) i + bi, where ai and bi are parameters, turns the denoising cost into a stochastic estimate of the contractive cost.”
Towards Deep Neural Network Architectures Robust to Adversarial Examples (2014, methodology): “(2012); Tang (2013). Another work inspired by the recursive nature of the human perceptual system is Deep Attention Selective Network (dasNet) Stollenga et al.”
Deep Learning of Representations: Looking Forward (2013, methodology): “…priors on the form of the model can be used to enhance disentangling, such as multiplicative interactions between the factors (Tenenbaum and Freeman, 2000; Desjardins et al., 2012) or orthogonality of the features derivative with respect to the input (Rifai et al., 2011b, 2012a; Sohn et al., 2013).”
Big Neural Networks Waste Capacity (2013, background): “3For speech recognition, Seide et al. (2011) report relative word error rates decreasing by about 30% on datasets of 309 hours 4http://www.image-net.org/challenges/LSVRC/2010/ ar X iv :1 30 1.”
Avoiding pathologies in very deep networks (2014, background): “We follow Rifai et al. (2011a) in characterizing the representational properties of a function by the singular value spectrum of the Jacobian.”
On the Expressive Power of Deep Architectures (2011, background): “Contractive autoencoders (Rifai et al., 2011) minimize a training criterion that is the sum of a reconstruction error and a “contraction penalty”, which encourages the learnt representation h(x) to be as invariant as possible to the input x, while still allowing to distinguish the training examples…”
A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU (2023, background): “Examples of regularized autoencoder models include Sparse Autoencoder (SAE) [ 61 ], Contractive Autoencoder (CAE) [ 62 ], and Denoising Autoencoder (DAE) [ 63 ].”
Laplacian Auto-Encoders: An explicit learning of nonlinear data manifold (2015, methodology): “RBM-binary : Restricted Boltzmann Machines [11]; AE: Basic auto-encoder [13]; AEþwd: Auto-encoder with weight-decay regularization [15]; DAE-g: Denoising auto-encoder with Gaussian noise [14]; DAE-b: Denoising auto-encoder with binary mask noise [14]; CAE: Contractive auto-encoder [15]; CAE-H: Contr”
Adaptive Noise Schedule for Denoising Autoencoder (2014, background): “Several variants of deep hierarchical representation composed of stacked autoencoders have been proposed [1], [4, 5, 6].”
Estimating crop yields with deep learning and remotely sensed data (2015, methodology): “After pre-training, back propagation is performed to obtain a better solution, because the spread of errors at the output layer is prevented [9].”
Foundations and Advances in Deep Learning (2014, methodology): “Furthermore, Rifai et al. (2011a) proposed to regularize, in addition to the Jacobian, the Hessian Hf approximated using the finite-difference method by modifying Eq.”
A Cross-Modality Learning Approach for Vessel Segmentation in Retinal Images (2016, methodology): “1) Theory of the Autoencoder: The autoencoder model [21] is often used to build deep networks.”
Estimating the Hessian by Back-propagating Curvature (2012, methodology): “It is also worth noting that this estimator underlies the Hessian norm estimation technique used in Rifai et al. (2011). That this is true is due to the equivalence between the stochastic finite-difference formulation used in that work and matrix-vector products with randomly drawn vectors.”