Importance Weighted Autoencoders

By Yuri Burda, Roger Baker Grosse, Ruslan Salakhutdinov. 2015

1,359 citations recorded by the source corpus.

The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It typically makes strong assumptions about posterior inference, for instance that the posterior distribution is approximately factorial, and that its parameters can be approximated with nonlinear regression from the observations. As we show empirically, the VAE objective can lead to overly simplified representations which fail to use the network's entire modeling capacity. We present the importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting. In the IWAE, the recognition network uses multiple samples to approximate the posterior, giving it increased flexibility to model complex posteriors which do not fit the VAE modeling assumptions. We show empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log-likelihood on density estimation benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Discrete Representation Learning (2017, background): “VIMCO [28] optimises a multi-sample objective [5], which speeds up convergence further by using multiple samples from the inference network.”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “1 under mild conditions [9]. as Bachman and Precup [3] introduced the idea of viewing IWAE auxiliary variable variational inference and Naesseth et al. [51], Cremer et al. [13], Domke and Sheldon [20] formalized the notion.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “Here, the complementary research into learning a covariance structure through a framework of normalizing flows [39, 52, 23] and the stacking of a hierarchy of latent variables [4, 37, 31, 50] have shown promising results.”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “Our results are reported using 16384 importance samples in our CIFAR experiment and 1 sample in our ImageNet experiments (Burda et al., 2015).”
Generalizing Hamiltonian Monte Carlo with Neural Networks (2017, background): “Recently, to improve upon well-known pitfalls like over-pruning (Burda et al., 2015) of the VAE, Hoffman (2017) proposed HMC-DLGM.”
Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “Table 1: Generative modeling results on the dynamically sampled binarized MNIST version used in previous publications (Burda et al., 2015).”
Density estimation using Real NVP (2016, background): “Still, the approximation in the inference process limits its ability to learn high dimensional deep representations, motivating recent work in improving approximate inference [42, 48, 55, 63, 10, 59, 34].”
Score-based Generative Modeling in Latent Space (2021, methodology): “Although this baseline achieves competitive NLL, its sample quality is behind our LSGM and the original SGM. Modeling binarized images: Next, we examine LSGM on dynamically binarized MNIST [93] and OMNIGLOT [74].”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, methodology): “For the VAE, we use the IWAE bound (Burda et al., 2015) with 10,000 posterior samples to estimate its likelihood.”
Diagnosing and Enhancing VAE Models (2019, background): “Unlike standard VAEs that are capable of self-calibration in some sense, with γ and elements of Σ z jointly pushing towards small values, γ may be diﬃcult to learn for WAE models because there is also a required weighting factor balancing the MMD (or GAN) loss.”
Unrolled Generative Adversarial Networks (2016, background): “…(Vincent et al., 2010), variational au-toencoders (Kingma & Welling, 2013; Rezende et al., 2014; Gregor et al., 2015; Kulkarni et al., 2015; Burda et al., 2015; Kingma et al., 2016), generative stochastic networks (Alain et al., 2015), diffusion probabilistic models (Sohl-Dickstein et al.,…”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “Lemma 9 doesn’t assume the form of q(x0:N ), thereby it can be applied to more general Gaussian models, such as multi-layer VAEs with Gaussian decoders (Rezende et al., 2014; Burda et al., 2015).”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, methodology): “This is provably more accurate than the VAE bound [5].”
Adversarial Autoencoders (2015, background): “For example, variational autoencoders (VAE) [Kingma and Welling, 2014, Rezende et al., 2014] or importance weighted autoencoders [Burda et al., 2015] use a recognition network to predict the posterior distribution over the latent variables, generative adversarial networks (GAN) [Goodfellow et al.,…”
An Introduction to Probabilistic Programming (2018, background): “Arguably the first example of this was the importance weighted autoencoder (Burda et al., 2016) which, effectively, uses an importance sampling inference engine during variational autoencoder training to both tighten the bound and minimize the variance of gradient estimates during training.”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, methodology): “For m = 1, Lm(θ, φ) becomes is the widely used evidence lower bound (ELBO, Hoffman et al., 2013) on log pθ(x), while for m > 1, it is known as the importance weighted bound (Burda et al., 2016).”
Variational Lossy Autoencoder (2016, neutral): “(Salimans et al., 2014) 81.94 DLGM 2hl + IWAE (Burda et al., 2015a) 82.90 Discrete VAE (Rolfe, 2016) 80.04 LVAE (Kaae Sønderby et al., 2016) 81.74 DRAW + VGP (Tran et al., 2015) 79.88 IAF VAE (Kingma et al., 2016) 79 As seen in Table 2,3,4, with the same set of hyperparameters tuned on statically ”