Resampled Priors for Variational Autoencoders

By Matthias Bauer, Andriy Mnih. 2018

122 citations recorded by the source corpus.

We propose Learned Accept/Reject Sampling (LARS), a method for constructing richer priors using rejection sampling with a learned acceptance function. This work is motivated by recent analyses of the VAE objective, which pointed out that commonly used simple priors can lead to underfitting. As the distribution induced by LARS involves an intractable normalizing constant, we show how to estimate it and its gradients efficiently. We demonstrate that LARS priors improve VAE performance on several standard datasets both when they are learned jointly with the rest of the model and when they are fitted to a pretrained model. Finally, we show that LARS can be combined with existing methods for defining flexible priors for an additional boost in performance.

Subfields: Artificial Intelligence, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling (2020, background): “Also, prior work on VAEs has shown that learned re-sampling or energy-based transformation of their priors can be effective [55, 56].”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, methodology): “In practice, Bauer and Mnih [5] estimate the normalizing constant using 1024 samples during training and 10 10 samples during evaluation.”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, background): “Improving sample quality by rejection sampling has been previously explored for GANs [1] as well as for VAEs [3] which combines a learned rejecting sampling proposal with the prior in order to reduce its gap with the aggregate posterior.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Furthermore, classifiers [54, 55, 56], adversarial methods [57], and other techniques [58, 59] have been used to define prior distributions implicitly.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Bauer and Mnih (2019) use the Real NVP architecture (Dinh et al., 2017) to parameterize a proposal distribution for RS since the coupling layers allow for fast density evaluation and sampling.”
Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition (2019, methodology): “Even though using the aggregate posterior as the prior is the objective in multiple recent works, it can be challenging in high dimensions, lead to over-fitting and often comes at the expense of additional hyper-parameters [47], [48], [49].”
Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection (2019, methodology): “…VAE architectures (e.g., using more sophisticated posteriors (Rezende & Mohamed, 2015), priors (Tomczak & Welling, 2017; Papamakarios et al., 2017; Bauer & Mnih, 2019), decoders, etc., which is all complementary); (ii) use alternative lower bounds to the marginal log-likelihood, such as the…”
NCP-VAE: Variational Autoencoders with Noise Contrastive Priors (2020, methodology): “Bauer & Mnih (2019) propose a reweighting factor similar to ours, but it is trained via importance sampling.”
Autoregressive Energy Machines (2019, background): “Bauer & Mnih (2018) propose an energy-based approach for increasing the ﬂexibility of VAE priors, in which a neural network energy function is used to mask a pre-speciﬁed proposal function.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, unknown): “Alternatively, the aggregate posterior can be approximated with a learned prior; this has been achieved with a learned rejection sampling procedure that transforms a base distribution [7].”
Latent Space Autoregression for Novelty Detection (2018, background): “This choice is coherent with recent works from the density estimation community [38, 6].”
Generative Latent Flow: A Framework for Non-adversarial Image Generation (2019, background): “In [33], the authors proposed to learn an extra rule that tells whether to accept or reject samples from p(z).”
A RAD approach to deep mixture models (2019, methodology): “It provides a more constrainted but less sample wasteful approach than rejection sampling (Bauer & Mnih, 2019) by taking into account the untransformed sample z before selecting the mixture component k .”
From Variational to Deterministic Autoencoders (2019, methodology): “Approaches to cope with the aggregated posterior mismatch involve fixing a more expressive form for p(z) [5,26] therefore altering the VAE objective and requiring considerable additional computational efforts.”
Learning Latent Space Energy-Based Prior Model (2020, methodology): “VampPrior [55] parameterizes the prior based on the posterior inference model, while [3] proposes to construct rich priors using rejection sampling with a learned acceptance function, both yielding improved performance on grey scale images.”