InfoVAE: Information Maximizing Variational Autoencoders

By Shengjia Zhao, Jiaming Song, Stefano Ermon. 2017

491 citations recorded by the source corpus.

A key advance in learning generative models is the use of amortized inference distributions that are jointly trained with the models. We find that existing training objectives for variational autoencoders can lead to inaccurate amortized inference distributions and, in some cases, improving the objective provably degrades the inference quality. In addition, it has been observed that variational autoencoders tend to ignore the latent variables when combined with a decoding distribution that is too flexible. We again identify the cause in existing training criteria and propose a new class of objectives (InfoVAE) that mitigate these problems. We show that our model can significantly improve the quality of the variational posterior and can make effective use of the latent features regardless of the flexibility of the decoding distribution. Through extensive qualitative and quantitative analyses, we demonstrate that our models outperform competing approaches on multiple performance metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior.”
Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems (2019, background): “and limitations of such models [32, 33, 34, 35, 36, 37], enhance the interpretability of their results [38, 39, 40], as well as establish formal connections with well studied topics in mathematics and statistics, including importance sampling [41, 42] and optimal transport [43, 44, 45].”
Isolating Sources of Disentanglement in VAEs (2018, background): “It is argued that a higher mutual information can lead to better disentanglement [6], and some have even proposed to completely drop the penalty on this term during optimization [17, 18].”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “VAE D KL ( q φ ( z | x ) (cid:107) p ( z )) VIB [9] VAE D KL ( q φ ( z | x ) (cid:107) p ( z )) O PixelGAN-AE[18] VAE − I q φ ( x ; z ) O InfoVAE [5] VAE D KL ( q φ ( z | x ) (cid:107) p ( z )) D KL ( q φ ( z ) (cid:107) p ( z )) Info. dropout [10] VAE D KL ( q φ ( z | x ) (cid:107) p ( z )) TC ( q…”
Wasserstein Auto-Encoders (2017, background): “Finally, [14] independently proposed a regularized auto-encoder objective similar to [4] and our (4) based on very diﬀerent motivations and arguments.”
Structured Disentangled Representations (2018, methodology): “This has prompted a number of approaches that modify the VAE objective by adding, removing, or altering the weight of individual terms [Kumar et al., 2017, Zhao et al., 2017, Gao et al., 2018, Achille and Soatto, 2018].”
Learning Controllable Fair Representations (2018, background): “However, as Lr +C1 is the VAE objective, their solutions does not prefer high mutual information between x and z (referred to as the “information preference” property (Chen et al., 2016; Zhao et al., 2017b,a, 2018)).”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, background): “We use 32-dimensional z and optimize ELBO objective with SGD for text and Adam (Kingma & Ba, 2015) for images.”
A deep generative model for gene expression profiles from single-cell RNA sequencing (2017, methodology): “This way, our procedure maximizes mutual information between zn and xn [18].”
A deep generative model for single-cell RNA sequencing with application to detecting differentially expressed genes (2017, methodology): “This way, our procedure maximizes mutual information between z n and x n [18].”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “Alternative discussions of the hyperparameter β are presented by Zhao et al. (2017); Higgins et al. (2017); Alemi et al. (2017); Achille & Soatto (2018), who show that it controls the amount of information in the latent variable, I ( x ; z ) .”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “Despite having similar form, the above objective is very diﬀerent from the objective function of β -VAE [30] in many ways: (1) our goal is to learn a diverse sampling distribution r ψ ( X | c ) for a pretrained generative model rather than learning the generative model itself; (2) The ﬁrst part in o”
Associative Compression Networks for Representation Learning (2018, background): “, 2015) or adding auxiliary objectives that reward more informative codes, for example by maximising the mutual information between the prior distribution and generated samples (Zhao et al., 2017)—a tactic that has been fruitfully applied to Generative Adversarial Networks (Chen et al.”
Unsupervised Brain Abnormality Detection Using High Fidelity Image Reconstruction Networks (2020, background): “, the limited expressiveness of the VAEs inference models, the injected noise, and unsuitable element-wise measures such as the reconstruction squared error (Larsen et al., 2016; Zhao et al., 2019).”
Auto-Encoding Total Correlation Explanation (2018, methodology): “12 requires a fully factorized form of the decoder qφ(x|z), unlike VAE where qφ(x|z) can be flexible.2 As pointed out by (Zhao et al., 2017a), if we choose to use a more expressive distribution family, such as PixelRNN/PixelCNN (Van Oord et al., 2016; Gulrajani et al., 2017) for the decoder in a…”
Deep Semi-Supervised Anomaly Detection (2019, background): “, 2018), generative modeling (Chen et al., 2016; Hoffman & Johnson, 2016; Zhao et al., 2017; Alemi et al., 2018), and unsupervised representation learning in general (Hjelm et al.”
Variational Template Machine for Data-to-Text Generation (2020, methodology): “As introduced by previous works (Chen et al., 2016; Zhao et al., 2017; 2018), adding mutual information term to ELBO could alleviate KL collapse effectively and improve the quality of variational posterior.”
Learning Hierarchical Priors in VAEs (2019, background): “…et al., 2016; Sønderby et al., 2016; Hig-gins et al., 2017; Rezende & Viola, 2018); and (iii) adding mutual-information based constrains or regularisers to incen-tivise a good correspondence between the data and the latent variables (Alemi et al., 2018; Zhao et al., 2017; Chen et al., 2016a).”