Deep Variational Information Bottleneck

By Alexander A. Alemi, Ian S. Fischer, Joshua V. Dillon. 2017

2,241 citations recorded by the source corpus.

We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Evaluating Lossy Compression Rates of Deep Generative Models (2020, methodology): “8 has been used in other algorithms such as the Blahut-Arimoto algorithm (Arimoto, 1972) or the variational information bottleneck algorithm (Alemi et al., 2016).”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “First, VAEs maximize the mutual information between the input and latent variables [29, 30], requiring the networks to retain the information content of the input data as much as possible.”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “For our experiments, we compare DIM against various unsupervised methods: Variational AutoEncoders (VAE, Kingma & Welling, 2013), β-VAE (Higgins et al., 2016; Alemi et al., 2016), Adversarial AutoEncoders (AAE, Makhzani et al.”
What makes for good views for contrastive learning (2020, background): “ those that share the minimal information necessary to perform well at the downstream task. This idea is related to the idea of minimal sufﬁcient statistics [59] and the Information Bottleneck theory [65, 2], which have been previously articulated in the representation learning literature. This prin”
Variational Autoencoders for Collaborative Filtering (2018, background): “6 has information-theoretic connections described in Alemi et al. [1].”
Invariant Representations without Adversarial Training (2018, methodology): “Instead of following Alemi et al.[3], we again use the approximation to KL[ q(z|x) ‖ q(z) ] from Eq.”
On the Fairness of Disentangled Representations (2019, background): “For the representation to be disentangled, the loss is enriched with a regularizer that encourages structure in the aggregate encoder distribution [2, 14, 13, 24, 36, 49, 65].”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, methodology): “Inspired by the variational information bottleneck theory [3], we rewrite the standard VAE objective in an equivalent form as follows: where Rec ( x ) denotes the reconstruction loss over x and β is a scalar.”
Towards a Definition of Disentangled Representations (2018, background): “An alternative to hard wiring inductive biases into the network architecture is to instead learn a representation that is faithful to the underlying data structure [76, 2, 3, 5, 4, 8, 69].”
Isolating Sources of Disentanglement in VAEs (2018, background): “For instance, it is found that such representations are more generalizable and robust against adversarial attacks [5].”
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (2018, background): “Past work [2, 4, 21] has demonstrated the usefulness of information-theoretic methods in such settings.”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “lar. Information theory. The information-theoretic view has gained recent interest in machine learning due to the information bottleneck (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Alemi et al., 2017) and usage in generative modelling (Chen et al., 2016; Hjelm et al., 2019). As a ”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “Adversarial robustness can be assessed through looking at the accuracy/likelihood of perturbed examples and the distance perturbed examples lie for each classiﬁer as well as the predictive entropy of the models on adversaries [3, 49, 37].”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, methodology): “By introducing a variational approximator, a popular technique from variational toolkit (Alemi et al., 2017), we can construct a lower bound on the mutual information term in Eq.”
Understanding disentangling in β-VAE (2018, background): “In the information bottleneck literature, Y would typically stand for a classiﬁcation task, however the formulation can be related to the auto-encoding objective too [2].”
Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, methodology): “The VAE model has also been applied to the learning of representations that are invariant to a certain source of variation (Alemi et al. 2017; Louizos et al. 2016; Edwards and Storkey 2016; Chen et al. 2017).”