Towards Deeper Understanding of Variational Autoencoding Models

By Shengjia Zhao, Jiaming Song, Stefano Ermon. 2017

169 citations recorded by the source corpus.

We propose a new family of optimization criteria for variational auto-encoding models, generalizing the standard evidence lower bound. We provide conditions under which they recover the data distribution and learn latent features, and formally show that common issues such as blurry samples and uninformative latent features arise when these conditions are not met. Based on these new insights, we propose a new sequential VAE model that can generate sharp samples on the LSUN image dataset based on pixel-wise reconstruction loss, and propose an optimization criterion that encourages unsupervised learning of informative latent features.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “Contrarily, we attribute it to the lack of overall model expressiveness for accurately modeling complex input distributions, as discussed in [59, 41].”
Autoregressive Quantile Networks for Generative Modeling (2018, background): “Zhao et al. (2017) argue that the Gaussian posterior p θ ( x z ) implies an overly simple model, which, when unable to perfectly ﬁt the data, is forced to average (thus creating blur), and is not incentivized by the KL towards an alternative notion of approximate solu-tion.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior.”
Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, background): “There are several papers that discuss the issue of latent variable collapse [10], [12], [13].”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “…models for real datasets (e.g., text, images, etc.) already work with a relatively simple and disentangled prior p ( z ) having no interaction among the latent dimensions (e.g., the standard Gaussian N (0 , I )) (Bowman et al., 2015; Miao et al., 2016; Hou et al., 2017; Zhao et al., 2017).”
Learning Controllable Fair Representations (2018, background): “However, as Lr +C1 is the VAE objective, their solutions does not prefer high mutual information between x and z (referred to as the “information preference” property (Chen et al., 2016; Zhao et al., 2017b,a, 2018)).”
Taming VAEs (2018, methodology): “,Similarly to [36] we study VAEs with supplementary constraints in addition to the ELBO objective function and we study its behaviour in the information plane as in [31].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “This blurriness has been attributed to the maximum likelihood objective itself and MSE reconstruction loss, however, there is evidence that limited approximation of the true posterior is the root cause [258]; with MSE causing highly non-Gaussian posteriors.”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, methodology): “For the image data, rather than generating directly in image space, we instead generate in the feature space of a pre-trained ResNet-101 (He et al., 2016), in order to avoid issues with blurry generations for complex image data (Zhao et al., 2017).”
Multimodal Generative Models for Scalable Weakly-Supervised Learning (2018, unknown): “(Though the reconstructed images still suffer from the same blurriness that VAEs do (Zhao et al., 2017).”
Domain Fingerprints for No-Reference Image Quality Assessment (2019, background): “3) Adversarial Loss: The VAE architecture tends to generate blurry samples [33], which would not be desirable for practical use.”
Semi-Amortized Variational Autoencoders (2018, background): “…are able to utilize a powerful generative model without experiencing the “posterior-collapse” phenomenon often observed in VAEs, wherein the variational posterior collapses to the prior and the generative model ignores the latent variable (Bowman et al., 2016; Chen et al., 2017; Zhao et al., 2017).”
Generalized ELBO with Constrained Optimization , GECO (2018, methodology): “Similarly to [25] we study VAEs with supplementary constraints in addition to the ELBO objective function and we study its behaviour in the information plane as in [20].”
Learning Hierarchical Features from Generative Models (2017, background): “More generally, as shown in (Zhao et al., 2017), even when LELBO is not globally optimized, optimizing LELBO encourages q(z`|z>`) and p(z`|z>`) to match.”
Avoiding Latent Variable Collapse With Generative Skip Models (2018, background): “Many papers describe latent variable collapse (Bowman et al., 2015; Sønderby et al., 2016; Kingma et al., 2016; Chen et al., 2016; Zhao et al., 2017; Yeung et al., 2017).”
From Variational to Deterministic Autoencoders (2019, background): “Many works have focused on diagnosing the VAE framework, the terms in its objective [1, 22, 60], and ultimately augmenting it to solve optimization issues [13, 39].”
Learning Hierarchical Features from Deep Generative Models (2017, background): “More generally, as shown in (Zhao et al., 2017), even when LELBO is not globally optimized, optimizing LELBO encourages q(z`|z>`) and p(z`|z>`) to match.”
A Unified Feature Disentangler for Multi-Domain Image Translation and Manipulation (2018, background): “Moreover, as noted in [29], the VAE architecture tends to generate blurry samples, which would not be desirable for practical uses.”
High-Fidelity Pluralistic Image Completion with Transformers (2021, background): “Nonetheless, extra distribution assumption would inevitably hurt the quality of generated contents [40].”
Advances in Variational Inference (2017, background): “In this case, in order to also satisfy the first term, the inference model places its probability mass to match the prior [227], failing to learn a”