Information Dropout: Learning Optimal Representations Through Noisy Computation

By Alessandro Achille, Stefano Soatto. 2016

462 citations recorded by the source corpus.

The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties. We show that this can be solved by adding a regularization term, which is in turn related to injecting multiplicative noise in the activations of a Deep Neural Network, a special case of which is the common practice of dropout. We show that our regularized loss function can be efficiently minimized using Information Dropout, a generalization of dropout rooted in information theoretic principles that automatically adapts to the data and can better exploit architectures of limited capacity. When the task is the reconstruction of the input, we show that our loss function yields a Variational Autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Finally, we prove that we can promote the creation of optimal disentangled representations simply by enforcing a factorized prior, a fact that has been observed empirically in recent work. Our experiments validate the theoretical intuitions behind our method, and we find that Information Dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “, 2016), InfoDrop (Achille and Soatto, 2018), and HEX (Wang et al.”
Deep Variational Information Bottleneck (2017, background): “Finally, concurrent with our work and most closely related Achille & Soatto (2016) propose and study a variational bound on the information bottleneck objective, from the perspective of variational dropout and demonstrate its utility in learning disentangled representations for variational…”
Opening the Black Box of Deep Neural Networks via Information (2017, neutral): “This is consistent with other recent suggestions on the role of noise in Deep Learning [Achille & Soatto (2016),Kadmon & Sompolinsky (2016)].”
Learning Robust Representations by Projecting Superficial Statistics Out (2018, methodology): “We also compare with the popular DG methods (DANN (Ganin et al., 2016)) and another method called information-dropout (Achille & Soatto, 2018).”
Disentangling by Factorising (2018, background): “Information Dropout and DIP-VAE did not have enough experimental details in the paper nor publicly available code to have their results reproduced and compared against.”
Towards a Definition of Disentangled Representations (2018, background): “An alternative to hard wiring inductive biases into the network architecture is to instead learn a representation that is faithful to the underlying data structure [76, 2, 3, 5, 4, 8, 69].”
Isolating Sources of Disentanglement in VAEs (2018, methodology): “In particular, Appendix B of [22] shows the existence of the total correlation in a similar objective with a flexible prior and assuming optimality q(z) = p(z).”
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (2018, background): “Past work [2, 4, 21] has demonstrated the usefulness of information-theoretic methods in such settings.”
Understanding disentangling in β-VAE (2018, background): “The β -VAE objective is closely related to the information bottleneck principle [40, 5, 1, 2]: where I ( · ; · ) stands for mutual information and β is a Lagrange multiplier.”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “Using the terminology of [10], the goal in supervised representation learning is to learn a minimal (in terms of code length) representation z of the data x that is sufﬁcient for a task y (in the sense that it contains enough information to predict y ).”
Structured Disentangled Representations (2018, methodology): “This has prompted a number of approaches that modify the VAE objective by adding, removing, or altering the weight of individual terms [Kumar et al., 2017, Zhao et al., 2017, Gao et al., 2018, Achille and Soatto, 2018].”
Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations (2017, background): “The beneﬁt of stochastic training has been explained from difference perspectives, such as Bayesian (Kingma et al., 2015) and information theory (Shwartz-Ziv & Tishby, 2017; Achille & Soatto, 2016).”
Emergence of Invariance and Disentanglement in Deep Representations (2017, methodology): “ty of the resulting representation. In parallel developments, the IB Lagrangian was used as a regularized loss function for learning representation, leading to new information theoretic regularizers (Achille and Soatto, 2018; Alemi et al., 2017a; Alemi et al., 2017b). In this paper, we introduce an ”
Compressing Neural Networks using the Variational Information Bottleneck (2018, background): “…Note that Gaussian noise has previously been multiplied with layer-wise activations as a path towards network regularization (Kingma et al., 2015; Achille & Soatto, 2018); however, these works are not concerned with network compression and other modeling details are signiﬁcantly diﬀerent than…”
Caveats for information bottleneck in deterministic scenarios (2018, unknown): “The mapping from inputs to hidden layer activity can be either deterministic, as traditionally done in neural networks, or stochastic, as used in some architectures (Alemi et al., 2016; Kolchinsky et al., 2017; Achille & Soatto, 2018) 2 .”
Direct Validation of the Information Bottleneck Principle for Deep Nets (2019, background): “Also, due to practical difficulties in estimating the MI between DNN layers, most works focus on stochastic nets trained in an end-to-end fashion [1, 2, 4, 8].”