An Information-Theoretic Analysis of Deep Latent-Variable Models

By Alexander A. Alemi, Ben Poole, Ian S. Fischer, Joshua V. Dillon, Rif A. Saurous, Kevin P. Murphy. 2017

86 citations recorded by the source corpus.

We present an information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference. This framework emphasizes the need to consider latent-variable models along two dimensions: the ability to reconstruct inputs (distortion) and the communication cost (rate). We derive the optimal frontier of generative models in the two-dimensional rate-distortion plane, and show how the standard evidence lower bound objective is insufficient to select between points along this frontier. However, by performing targeted optimization to learn generative models with different rates, we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable. Through experiments on MNIST and Omniglot with a variety of architectures, we show how our framework sheds light on many recent proposed extensions to the variational autoencoder family.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards a Definition of Disentangled Representations (2018, background): “An alternative to hard wiring inductive biases into the network architecture is to instead learn a representation that is faithful to the underlying data structure [76, 2, 3, 5, 4, 8, 69].”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “As a consequence, the estimation of mutual information (Barber & Agakov, 2003; Alemi et al., 2018; Achille & Soatto, 2018; Belghazi et al., 2018) has attracted growing attention.”
VAE with a VampPrior (2017, methodology): “Additionally, the authors of [2] proposed to use a weighted version of the VampPrior: where w 1 , . . . , w K are trainable parameters such that ∀ k w k ≥ 0 and k w k = 1 .”
Distribution Matching in Variational Inference (2018, methodology): “Alemi et al. (2017) show that simple decoders have a smaller KL term but have high reconstruction error, whereas complex decoders are better suited for reconstruction.”
Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs (2019, background): “This disentangling-distortion trade-off is a recapitulation of the rate-distortion trade-off [Alemi et al., 2017] and can be seen first-hand in Figure 4.”
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music (2018, background): “The presence of these terms suggests a trade-off between the quality of samples and reconstructions–or equivalently, be-tween the rate (amount of information encoded in q λ ( z | x ) ) and distortion (data likelihood) (Alemi et al., 2017).”
InfoVAE: Balancing Learning and Inference in Variational Autoencoders (2019, background): “Relation to Information Theoretical Analysis: Several other lines of work analyze mutual information in latent variable generative models from different perspectives, including (Zhao, Song, and Ermon 2018; Alemi et al. 2017).”
Isolating Sources of Disentanglement in Variational Autoencoders (2018, background): “Such reasoning resonates with the observed behavior that β is a hyperparameter that requires careful tuning and that large β values typically fail Isolating Sources of Disentanglement in Variational Autoencoders in practice (Higgins et al., 2017; Alemi et al., 2017b).”
Generalization in Generation: A closer look at Exposure Bias (2019, background): “In addition, a whole body of work has discussed the implications of optimizing only a bound to the log-likelihood (Alemi et al., 2017) and the implications of re-weighting the information-theoretic quantities inside the bound (Higgins et al., 2017; Rainforth et al., 2018).”
GILBO: One Metric to Measure Them All (2018, methodology): “The GILBO contrasts with the representational mutual information of VAE s deﬁned by the data and encoder, which motivates VAE objectives (Alemi et al., 2017).”
A Lagrangian Perspective on Latent Variable Generative Models (2018, background): “In addition we also denote identical bounds for Ipθ as Ipθ and Ipθ Similar bounds for mutual information have been discussed in (Alemi et al., 2017).”