Contractive Auto-Encoders: Explicit Invariance During Feature Extraction

By Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, Yoshua Bengio. 2011

1,581 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “Autoencoders of this form have been successfully used for learning deep representations in neural networks (Rifai et al., 2011).”
Pylearn2: a machine learning research library (2013, methodology): “Pylearn2 has implementations of several models that were de v loped at LISA, including denoising auto-encoders (DAEs) [53], contractive auto-encoders (CAEs) [42] including higher-order CAEs [43], spike-and-slab RBMs (ssRBMs) [11] including ssR BMs with pooling [12], reconstruction sampling autoencod”
Generalized Denoising Auto-Encoders as Generative Models (2013, methodology): “The last step in this development (Alain and Bengio, 2013) generalized the result from Vincent (2011) by showing that when a denoising auto-encoder (or a contractive auto-encoder with the contraction on the whole encode/decode reconstruction function) is trained with small Gaussian corruption and…”
Better Mixing via Deep Representations (2012, background): “The CAE is trained to minimize a reconstruction loss (cross-entropy here) plus a contractive regularization penalty α||∂f(x)∂x || 2 F (which is the sum of the elements of the Jacobian matrix).”
A Generative Process for Contractive Auto-Encoders (2012, methodology): “To interpret the CAE from this perspective, as is done in Rifai et al. (2011b), it is useful to think of an au-toencoder as projecting an input point x onto a low dimensional manifold, and of the hidden representation as the coordinates of that projection in a coordinate system within the manifold.”
GAN Inversion: A Survey (2021, methodology): “Different from previous autoencoder models [169], [170] mapping the input x into a fixed vector, VAEs [158] map x into a latent vector z with prior distribution pθ(z), parameterized by θ.”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “We used the contractive penalty (found in Mescheder et al. (2018) but ﬁrst introduced in contractive autoencoders (Rifai et al., 2011)) on the encoder, gradient clipping on the discriminators, and no regularization on the generator.”
Generative Moment Matching Networks (2015, neutral): “This is analogous to the motivation behind contractive and denoising auto-encoders (Rifai et al., 2011; Vin-cent et al., 2008).”
Representation Learning: A Review and New Perspectives (2012, methodology): “7.2.3 Contractive Auto-Encoders Contractive Auto-Encoders (CAE), proposed by Rifai et al. (2011a), follow up on Denoising Auto-Encoders (DAE) and share a similar motivation of learning robust representations.”
Energy-based Generative Adversarial Network (2016, methodology): “Recently, a method called “minibatch discrimination” (MBD) (Salimans et al., 2016) has been proposed to avoid such triviality. The idea is that when training a GAN, instead of having the discriminator only see one image per iteration, giving it the access of an entire batch. Emitting overlapped patt”
Stacked What-Where Auto-encoders (2015, neutral): “model CIFAR-10 CIFAR-100 All-Convnet (Springenberg et al. (2014)) 92.”
Marginalized Denoising Autoencoders for Domain Adaptation (2012, background): “Various forms of au-toencoders have been developed in the deep learning literature (Rumelhart et al., 1986; Baldi & Hornik, 1989; Kavukcuoglu et al., 2009; Lee et al., 2009; Vincent et al., 2008; Rifai et al., 2011).”
Deep learning in neural networks: An overview (2014, background): “earning rate adjustments during stochastic gradient descent (Cho et al., 2013; Cho, 2014), and through Tikhonovtype (Tikhonov et al., 1977) regularization of RBMs (Cho et al., 2012). Contractive AEs (Rifai et al., 2011) discourage hidden unit perturbations in response to input perturbations, similar”
Towards Deep Neural Network Architectures Robust to Adversarial Examples (2014, methodology): “Contractive autoencoder (CAE) is a variant of an autoencoder (AE) with an additional penalty for minimizing the squared norm of the Jacobian of the hidden representation with respect to input data Rifai et al. (2011b).”
Adversarial Vulnerability of Neural Networks Increases With Input Dimension (2018, methodology): “Penalizing network-gradients is also at the heart of contractive auto-encoders as proposed by [RVM11], where it is used to regularize the encoder-features.”
Robust Large Margin Deep Neural Networks (2016, background): “The work in [27] is related to ours in the sense that it proposes to regularize auto-encoders by constraining the Frobenious norm of the encoder’s Jacobian matrix.”
Distilling Knowledge from Deep Networks with Applications to Healthcare Domain (2015, methodology): “It has been successfully used for feature extraction or as a pre-training step for a neural network [37, 19].”
A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification (2018, neutral): “Therefore, they are picked as the peer competitors of the proposed algorithm, which are CAE-2 [13], TIRBM [15], PGBM+DN1 [5], ScatNet-2 [2], RandNet-2 [3], PCANet-2 (softmax) [3], LDANet-2 [3], SVM+RBF [8], SVM+Poly [8], NNet [8], SAA-3 [8] and DBN-3 [8].”
Relational autoencoder for feature extraction (2017, background): “[22] proposed a Contractive Autoencoder (CAE).”
Learning Robust Representations by Projecting Superficial Statistics Out (2018, methodology): “We compared the performance of HEX/ADV with several methods tested on this data including CAE (Rifai et al., 2011), MTAE (Ghifary et al.”