Emergence of invariance and disentangling in deep representations

By Alessandro Achille, Stefano Soatto. 2017

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “There are various definitions for disentanglement [50, 46, 1, 6, 18], but a common goal is a latent space that consists of linear subspaces, each of which controls one factor of variation.”
Invariant Representations without Adversarial Training (2018, background): “Achille and Soatto [1] directly utilize these results, providing the same criterion and relaxation for invariance we will use here, minimal mutual information between the representation z and the covariate c .”
Isolating Sources of Disentanglement in VAEs (2018, background): “While the definition of disentanglement is open to debate, many believe a factorial representation, one with statistically independent variables, is a good starting point [1, 2, 3].”
Critical Learning Periods in Deep Neural Networks (2017, background): “Understanding the connection between optimization and representation is a subject of active investigation (16–18).”
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (2018, background): “This is because the implicitly learnt representations often overfit to the training task by discarding information that is irrelevant to the current task but may be required for solving future tasks [1, 2, 3, 51, 22].”
i-RevNet: Deep Invertible Networks (2018, background): “It is widely believed that this process is based on progressively discarding uninformative variability about the input with respect to the problem at hand (Dosovitskiy & Brox, 2016; Mahendran & Vedaldi, 2016; Shwartz-Ziv & Tishby, 2017; Achille & Soatto, 2017).”
A Bayesian Perspective on Generalization and Stochastic Gradient Descent (2017, neutral): “The Bayesian evidence can be reframed in the language of information theory, whereby Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data (Hinton & Van Camp, 1993; Achille & Soatto, 2017; Shwartz-Ziv & Tishby, 2017).”
The Description Length of Deep Learning models (2018, background): “…in the weights: there is no mutual information between inputs and outputs. how the hidden representations in a neural network compress the inputs while preserving the mutual information between inputs and outputs (Tishby and Zaslavsky, 2015; Shwartz-Ziv and Tishby, 2017; Achille and Soatto, 2017).”
On the importance of single directions for generalization (2018, background): “For example, Achille & Soatto (2017) demonstrated that networks trained on randomized labels store more information in their weights than those trained on unmodﬁed labels.”
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks (2017, background): “Minimizing this functional, explicitly, has been shown to lead to invariant representations (Achille and Soatto, 2017).”
Stochastic Mirror Descent on Overparameterized Nonlinear Models (2019, background): “There have been many efforts in the past few years to study deep learning from an optimization perspective, e.g., (Achille & Soatto, 2017; Chaudhari & Soatto, 2018; Shwartz-Ziv & Tishby, 2017; Allen-Zhu et al., 2019; Oymak & Soltanolkotabi, 2019; Azizan & Hassibi, 2019; Ma et al., 2018b; Du et al.,…”
Hierarchical quantum classifiers (2018, background): “Some kinds of noise can even help with convergence and generalization [36, 37].”
A3T: Adversarially Augmented Adversarial Training (2018, background): “Achille and Soatto (2017) takes the view from information theory to a regularizing term added to the classiﬁer loss, which ﬁlters out the information of a given class of ‘nuisances’, i.e variations of the inputs that are irrelevant to the task.”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy (2017, background): “For an appropriate range of β , Achille and Soatto (2017) connect the objective with so-called linear PAC-Bayes bounds of Catoni (see (McAllester, 2013)).”
Factorised spatial representation learning: application in semi-supervised myocardial segmentation (2018, background): “layers, but invariance to more complex factors is achieved by the learning process, and specically encouraged by regularisers (explicit regularisation) or data augmentation (implicit regularisation) [1]. At a high level the aim is to keep relevant and discard irrelevant information, however which in”
Initialization matters: Orthogonal Predictive State Recurrent Neural Networks (2018, background): “Even though these approaches are only loosely related to our work, there is a common denominator: orthogonality, whether applied in our context or the aforementioned ones, seems to to be responsible for disentangling in (deep) representations (Achille & Soatto, 2017).”
A Walk with SGD (2018, background): “This may be explained by Neyshabur et al. (2017); Achille and Soatto (2017) that discuss that minima that are both wide and have small norm may explain generalization in over-parametrized deep networks.”
Auto-Encoding Total Correlation Explanation (2018, methodology): “Total correlation or statistical in-dependence is often used to characterize disentanglement in recent literature on learning representations (Dinh et al., 2014; Achille & Soatto, 2017).”
Entropy-Constrained Training of Deep Neural Networks (2018, background): “Other related work have also focused on designing regularizers that lower the implicit information entailed in the networks parameters, based on minimizing a variational lower bound objective [19], [20], [35].”