Measuring Invariances in Deep Networks

By Ian J. Goodfellow, Quoc V. Le, Andrew M. Saxe, Honglak Lee, Andrew Y. Ng. 2009

480 citations recorded by the source corpus.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Intriguing properties of neural networks (2013, background): “So far, unit-level inspection methods had relatively little utility beyond conﬁrming certain intuitions regarding the complexity of the representations learned by a deep neural network [6, 13, 7, 4].”
Better Mixing via Deep Representations (2012, background): “Several observations suggest that some deep learning algorithms indeed help to disentangle the underlying factors of variation (Goodfellow et al., 2009; Glorot et al., 2011).”
A Generative Process for Contractive Auto-Encoders (2012, methodology): “In order to validate that the proposed criterion does learn features that are more invariant to transformations of interest, we deﬁne the following average normalized sensitivity score, following the ideas put forward in Goodfellow et al. (2009).”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “, 2016) while being independent from the task at hand (Goodfellow et al., 2009; Lenc and Vedaldi, 2015).”
Making Convolutional Networks Shift-Invariant Again (2019, background): “…or output changes (or robustness to changes) in response to manually generated perturbations to the input, such as image transformations (Goodfellow et al., 2009; Lenc & Vedaldi, 2015; Azulay & Weiss, 2019), geometric transforms (Fawzi & Frossard, 2015; Ruderman et al., 2018), and CG…”
Representation Learning: A Review and New Perspectives (2012, background): “…be achieved by penalizing the hidden unit biases (making these additive offset parameters more negative) (Ranzato et al., 2007; Lee et al., 2008; Goodfellow et al., 2009; Larochelle and Bengio, 2008) or by directly penalizing the output of the hidden unit activations (making them closer to…”
Building high-level features using large scale unsupervised learning (2011, positive): “t consisting of faces and random distractors, we discovered a feature that is highly selective for faces. This further conrms the power of deep networks in learning invariant feature representations [7, 8]. 2 Dataset constructions Our dataset is constructed by sampling frames from 10 million YouTube”
How to Construct Deep Recurrent Neural Networks (2013, background): “Previous work has shown that higher-level representations of deep networks tend to better disentangle the underlying factors of variation than the original input (Goodfellow et al. , 2009; Glorot et al. , 2011b) and ﬂatten the manifolds near which the data concentrate (Bengio et al. , 2013).”
Explaining nonlinear classification decisions with deep Taylor decomposition (2015, background): “transformations of the data [31], the role of particular neurons [32], or its global decision structure [33, 34].”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “[1, 40, 26, 2, 42, 24, 47], since they should contain all the information present in the observations in a compact and interpretable structure [1, 21, 6] while being independent from the task at hand [12,28]. Disentangled representations may be useful for (semi-)supervised learning of downstream tas”
On the Fairness of Disentangled Representations (2019, background): “Indeed, isolating each independent factor of variation into the independent components of a representation vector should make it both interpretable and simplify downstream prediction tasks [6, 7, 29, 35, 56, 58, 60, 74, 82, 87, 89, 90, 1, 28].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “ng on the high dimensional x. Indeed, they should contain all the information present in x in a compact and interpretable structure [4, 29, 8]. Furthermore, they are independent from the task at hand [17, 34]. Therefore, they would be useful for (semi-)supervised learning of downstream tasks, transf”
Disentangling by Factorising (2018, background): “Invariance and equivariance are frequently considered to be desirable properties of representations in the literature (Goodfellow et al., 2009; Kivinen & Williams, 2011; Lenc & Vedaldi, 2015).”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “As discussed in Goodfellow et al. (2009) sparse representations may be advantageous because they encourage representations that disentangle the underlying factors of representation.”
Learning the Irreducible Representations of Commutative Lie Groups (2014, neutral): “Here we focus on the notions of invariance and disentangling, leaving the search for meaning for future work.”
DeepMPC: Learning Deep Latent Features for Model Predictive Control (2015, methodology): “Phase 1: Unsupervised Pre-Training: In order to obtain a good initial set of features for l, we apply an unsupervised learning algorithm similar to the sparse auto-encoder algorithm [16] to train the non-recurrent parameters of the TRUs.”