Distribution Matching in Variational Inference

By Mihaela Rosca, Balaji Lakshminarayanan, Shakir Mohamed. 2018

106 citations recorded by the source corpus.

With the increasingly widespread deployment of generative models, there is a mounting need for a deeper understanding of their behaviors and limitations. In this paper, we expose the limitations of Variational Autoencoders (VAEs), which consistently fail to learn marginal distributions in both latent and visible spaces. We show this to be a consequence of learning by matching conditional distributions, and the limitations of explicit model and posterior distributions. It is popular to consider Generative Adversarial Networks (GANs) as a means of overcoming these limitations, leading to hybrids of VAEs and GANs. We perform a large-scale evaluation of several VAE-GAN hybrids and analyze the implications of class probability estimation for learning distributions. While promising, we conclude that at present, VAE-GAN hybrids have limited applicability: they are harder to scale, evaluate, and use for inference compared to VAEs; and they do not improve over the generation quality of GANs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “A mismatch could create “holes” [79, 43, 3] in the prior that the aggregate posterior fails to cover during training, resulting in worse sample quality, as many latent variables used during generation are likely to never have been trained on.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “While VAE is an appealing choice for this task, balancing between retaining rich information in the latent code and maintaining the sampling quality in VAE is hard [41,42,45,52].”
Disentangling by Factorising (2018, background): “From Figure 7, observe that the discriminator is consistently underestimating the true TC, also confirmed in (Rosca et al., 2018).”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “However, this brings forth another well-known issue with VAEs: distribution mismatch [43; 46; 47].”
Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems (2019, background): “[33] revealed some drawbacks in the original formulation of Kingma and Welling [22] are attributed to the form of the variational objective in Eq.”
Do Deep Generative Models Know What They Don't Know? (2018, neutral): “See Rosca et al. (2018) for descriptions of the precise versions we use.”
Taming VAEs (2018, methodology): “…implementations of ConvDraw [5] (both in the conditional and unconditional case) and a VAE+NVP model that uses a convolutional decoder similar to [28] and a fully connected conditional NVP [58] model as the encoder density so that we can approximate high-capacity encoders. and ELBO with a hand…”
Resampled Priors for Variational Autoencoders (2018, methodology): “More recently, both the role of the prior and its mismatch to the aggregate variational posterior have been investigated in detail (Hoffman and Johnson, 2016; Rosca et al., 2018).”
Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “We did not compare with AVB since the estimated log marginal likelihood of AVB with high-dimensional datasets such as images is not accurate (Rosca, Lakshminarayanan, and Mohamed 2018).”
An Introduction to Variational Autoencoders (2019, background): “As such many hybrid models have been proposed to try to represent the best of both worlds (Dumoulin et al., 2017; Grover et al., 2018; Rosca et al., 2018).”
Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality (2019, background): “[48]’s VAE architecture—attempting to replicate training and evaluation as closely as possible.”
Detecting Out-of-Distribution Inputs to Deep Generative Models Using a Test for Typicality (2019, methodology): “We use the same three generative models as they did—Glow [22], PixelCNN [38], and Rosca et al. [30]’s VAE architecture—attempting to replicate training and evaluation as closely as possible.”
Generalized ELBO with Constrained Optimization , GECO (2018, methodology): “[18, 33, 13, 16]) can also be mathematically interpreted as constraints which bound the values of the likelihood from above.”
Pioneer Networks: Progressively Growing Generative Autoencoder (2018, background): “Often, there is a trade-off between the two [27].”
Understanding Uncertainty in Bayesian Neural Networks (2019, background): “The issue of distribution mismatch in VAEs is, in part, responsible for these models producing worse quality samples than other DGMs and has been studied at length (Antoran and Miguel, 2019; Rosca et al., 2018).”
Artificial intelligence in the creative industries: a review (2020, background): “However, the results of this have not yet demonstrated significant improvement in terms of overall performance (Rosca et al. 2019), remaining an ongoing research topic.”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, background): “Although there has been tremendous progress in improving the expressivity of the approximate posterior, several studies have observed that VAE priors fail to match the aggregate (approximate) posterior (Rosca et al., 2018; Hoffman & Johnson, 2016).”
From Variational to Deterministic Autoencoders (2019, result): “1, issue i) is compromising the VAE framework in any case in practice as reported in several works [14, 22, 42].”
On Maximum Likelihood Training of Score-Based Generative Models (2021, unknown): “…normalizing ﬂows [Papa-makarios et al., 2019], which may fail to map data to their chosen base distribution, or variational autoencoders [Rezende et al., 2014; Kingma and Welling, 2014], whose aggregate posterior may deviate arbitrarily from the prior [Hoﬀman and Johnson, 2016; Rosca et al., 2019].”