Diagnosing and Enhancing VAE Models

By Bin Dai, David Paul Wipf. 2019

441 citations recorded by the source corpus.

Although variational autoencoders (VAEs) represent a widely influential deep generative model, many aspects of the underlying energy function remain poorly understood. In particular, it is commonly believed that Gaussian encoder/decoder assumptions reduce the effectiveness of VAEs in generating realistic samples. In this regard, we rigorously analyze the VAE objective, differentiating situations where this belief is and is not actually true. We then leverage the corresponding insights to develop a simple VAE enhancement that requires no additional hyperparameters or sensitive tuning. Quantitatively, this proposal produces crisp samples and stable FID scores that are actually competitive with a variety of GAN models, all while retaining desirable attributes of the original VAE architecture. A shorter version of this work will appear in the ICLR 2019 conference proceedings (Dai and Wipf, 2019). The code for our model is available at this https URL TwoStageVAE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Two-Stage Image Synthesis To mitigate the shortcomings of individual generative approaches, a lot of research [11,23,65,68,97,99] has gone into combining the strengths of different methods into more efficient and performant models via a two stage approach.”
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “Most works [55, 18, 53] learn a discrete representation which is subsequently modeled autoregressively but approaches using continuous representations in combination with VAEs [10], or normalizing ﬂows [1, 57, 17], exist too.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Other methods adopt a “two-stage” approach [23], which fits a generative model over the latent space of autoencoders [96, 75, 26, 74].”
Score-based Generative Modeling in Latent Space (2021, methodology): “In two-stage training, a separate generative model is trained in latent space as a new prior after training the VAE itself [60, 61, 62, 63, 64, 10].”
Taming Transformers for High-Resolution Image Synthesis (2020, background): “[13] demonstrated both theoretical and empirical evidence on the advantages of first learning a data representation with a Variational Autoencoder (VAE) [34, 54], and then again learning its distribution with a VAE.”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, background): “Dai & Wipf (2019) recently showed that when a VAE is trained on data distributed on a K -dimensional manifold embedded in R D , a VAE will only activate K dimensions in its latent space, meaning that the VAE will require fewer layers unless the manifold dimension is D , which is unlikely to be the…”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “Previous work in this direction learns the latent structure of pre-trained VAEs (Dai & Wipf, 2019; Xiao et al., 2019; Ghosh et al., 2020), and sampling from learned latent distributions improves sample quality.”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, neutral): “Several recent works approached this issue [5, 40, 8, 1, 12, 31, 9].”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, methodology): “We refer the reader to (Dai & Wipf, 2019) for a detailed analysis.”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, methodology): “train on the same datasets with losses as in [26]. Tab.2presents mean and std of FID scores over three trials with 10K generated images. We significantly improve over state-of-the-art FID reported in [4]. Our approach can utilize a learned similarity metric similar to GANs, which enables them to pro”
Further Analysis of Outlier Detection with Deep Generative Models (2020, unknown): “6 For the VAE-based tests, we use an architecture similar to [19], and vary the latent dimension nz as it may have an influence on the likelihood-based outlier test.”
Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning (2020, methodology): “However, InfoGAN is known to suffer from training instability [48], and there is no guarantee that each latent variable is semantically meaningful [30, 7].”
Learning Latent Space Energy-Based Prior Model (2020, methodology): “Bearing with the feasible cost our method leads to performance improving over strong baselines such as 2sVAE [11] and RAE [16] on image and text modeling and anomaly detection.”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “…we will ﬁnd experimentally that a simpler parameterization, in which the covariance matrix is speciﬁed with a single shared (Kingma et al., 2016; Dai & Wipf, 2019; Edwards & Storkey, 2016; Rezende & Viola, 2018) parameter σ as Σ = σI often works better in practice: The parameter σ can be…”
Planning with Goal-Conditioned Policies (2019, background): “In fact, Dai and Wipf [9] showed that a VAE with a Gaussian prior can always recover the true manifold, making this choice for latent-variable model particularly appealing.”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “This mirrors many public VAE implementations where σ 2 is ﬁxed to 1 throughout training (also observed by Dai and Wipf [13]), however, our linear analysis suggests that this is suboptimal.”
From Variational to Deterministic Autoencoders (2019, background): “Estimating the latent space of a VAE with a second VAE [13] reintroduces many of the optimization shortcomings discussed for VAEs and is much more expensive in practice compared to fitting a simple qδ(z) after training.”
A Generative Approach Towards Improved Robotic Detection of Marine Litter (2019, background): “Among recent VAE enhancements, only the two-stage VAE [24] focuses on improving visual quality scores rather than improving loglikelihood scores.”