VAE with a VampPrior

By J. Tomczak, Max Welling. 2017

724 citations recorded by the source corpus.

Many different methods to train deep generative models have been introduced in the past. In this paper, we propose to extend the variational auto-encoder (VAE) framework with a new type of prior which we call "Variational Mixture of Posteriors" prior, or VampPrior for short. The VampPrior consists of a mixture distribution (e.g., a mixture of Gaussians) with components given by variational posteriors conditioned on learnable pseudo-inputs. We further extend this prior to a two layer hierarchical model and show that this architecture with a coupled prior and posterior, learns significantly better models. The model also avoids the usual local optima issues related to useless latent dimensions that plague VAEs. We provide empirical studies on six datasets, namely, static and binary MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces and Histopathology patches, and show that applying the hierarchical VampPrior delivers state-of-the-art results on all datasets in the unsupervised permutation invariant setting and the best results or comparable to SOTA methods for the approach with convolutional networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Furthermore, classifiers [54, 55, 56], adversarial methods [57], and other techniques [58, 59] have been used to define prior distributions implicitly.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, neutral): “90 VampPrior [64] 78.”
Diagnosing and Enhancing VAE Models (2019, methodology): “, 2018) replace N (z|0, I) with a parameterized class of prior distributions such that there exist more flexible pathways for pushing p(z) and qφ(z) closer together; a VAE-like objective is used for this purpose in (Tomczak and Welling, 2018), while (Zhao et al.”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “It has been previously noted that the use of overly simplistic priors like standard normal Gaussians in VAE models [25] can lead to overregularization [53].”
Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, methodology): “As methods to mitigate these effects are still areas of active research (see references in He et al. (2019) and Tomczak & Welling (2018)), we also train the RNF without the KL regularisation terms – which can be interpreted as placing a ﬂat prior over the latent state and removing constants from…”
BooVAE: Boosting Approach for Continual Learning of VAE (2019, methodology): “Following (Tomczak & Welling, 2018), we select the parametric family R of the components h as learnable pseudo-inputs u to the encoder: h ( z )= q φ ( z u ) .”
Lifelong Generative Modeling (2017, neutral): “Posterior collapse occurs when training a VAE with a powerful decoder model such as a PixelCNN++ [127] or RNN [21, 40].”
Adversarial Latent Autoencoders (2020, background): “For the latent space instead, the common practice is to set a desired target latent distribution, and then the encoder is trained to match it either by minimizing a divergence type of similarity [28, 41, 31, 47, 48], or by setting up an adversarial game [35, 6, 9, 49, 20].”
Taming VAEs (2018, background): “It has been observed empirically that VAEs with simple posterior models have a tendency to ignore some of the latent-variables (latent-collapse) [19, 20] and produce blurred reconstructions [20, 4].”
Resampled Priors for Variational Autoencoders (2018, background): “08 RealNVP + Lars high capacity a(z) N (0, 1) + Lars high capacity a(z): MLP[10-10] N (0, 1) + Lars low capacity a(z): MLP[10]”
DVAE++: Discrete Variational Autoencoders with Overlapping Transformations (2018, background): “DVAE++ is related to VAEs with mixture priors (Makhzani et al., 2015; Tomczak & Welling, 2017).”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “…et al., 2018), deploying less powerful de-coders (Yang et al., 2017; Semeniuta et al., 2017), using lossy input (Chen et al., 2017), utilizing different latent variable connections (Dieng et al., 2017; 2018; Park et al., 2018), or changing the prior (Tomczak & Welling, 2018; Xu & Durrett, 2018).”
Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “These image datasets are useful for measuring the density estimation performance, which are used in (Tomczak and Welling 2018).”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, background): “Many researches around VAE models deal with building more expressive distributions for all its components [4, 39, 40, 30, 31].”
Learning Latent Space Energy-Based Prior Model (2020, background): “However, such assumption may cause ineffective generator learning as observed in [11, 56].”
An Information-Theoretic Analysis of Deep Latent-Variable Models (2017, methodology): “However, a number of papers have observed that VAEs trained with powerful decoders can learn to ignore the latent variables (Chen et al., 2017; Tomczak & Welling, 2017; Bowman et al., 2016).”