Reinterpreting Importance-Weighted Autoencoders

By Chris Cremer, Quaid D. Morris, David Kristjanson Duvenaud. 2017

103 citations recorded by the source corpus.

The standard interpretation of importance-weighted autoencoders is that they maximize a tighter lower bound on the marginal likelihood than the standard evidence lower bound. We give an alternate interpretation of this procedure: that it optimizes the standard variational lower bound, but using a more complex distribution. We formally derive this result, present a tighter lower bound, and visualize the implicit importance-weighted distribution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “1 under mild conditions [9]. as Bachman and Precup [3] introduced the idea of viewing IWAE auxiliary variable variational inference and Naesseth et al. [51], Cremer et al. [13], Domke and Sheldon [20] formalized the notion.”
Inference Suboptimality in Variational Autoencoders (2018, methodology): “As shown by Bachman & Precup (2015) and Cremer et al. (2017), the IWAE bound can be seen as using the VAE bound but with an importance weighted q distribution.”
Importance Weighting and Variational Inference (2018, background): “Previous work has connected IWVI and NIS by showing that the importance weighted ELBO is a lower bound of the ELBO applied to the NIS distribution [6, 16, 2].”
Variational Sequential Monte Carlo (2017, methodology): “However, VIS provides a new interpretation that enables a more accurate variational approximation; this relates to another interpretation of IWAE by Cremer et al. [2017], Bachman and Precup [2015].”
Divide and Couple: Using Monte Carlo Variational Objectives for Posterior Approximation (2019, background): “If we optimize a variational objective obtained from a different estimator, does this still correspond to minimizing some KL-divergence? It has been shown [10, 20, 11] that maximizing the IWAE objective corresponds to minimizing (an upper bound to) KL[qIS(z)kp(z|x)], where qIS is a version of q that”
Advances in Variational Inference (2017, background): “encoder (IWAE) which was originally proposed to tighten the variational bound [25] and can be reinterpreted to sample from a more flexible distribution [31].”
Tighter Variational Bounds are Not Necessarily Better (2018, result): “As such, this result, which complements those of (Cremer et al., 2017), suggests that increasing K provides a preferable target in terms of the direction of the true inference network gradients.”
Learning Hierarchical Priors in VAEs (2019, background): “Since a diagonal Gaussian qΦ(ζ|z) is not flexible enough to capture the (true) posterior of the second stochastic layer, we propose to enhance it by using an importance-weighted bound (Cremer et al., 2017) (alternatively, one could use, for example, normalising flows (Rezende & Mohamed, 2015)).”
Amortized Inference Regularization (2018, methodology): “Analysis by [22] allows us to rewrite it as a regularized maximum likelihood objective”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, background): “To see this, consider the form of the gradient estimator of this objective with respect to the variational posterior parameters Φ (Burda et al., 2015; Cremer et al., 2017), where , and ¯ w k = w k Kj =1 w j , indicating that the gradient weights samples by their relative importance ¯ w k .”
Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference (2018, background): “sense in light of the work of Cremer et al. (2017). Interestingly the effect is stronger for JVI-1.”
MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets (2019, methodology): “To compare models, we evaluate estimates of their test log-likelihood obtained using importance sampling with 5000 samples and an inference network refitted on the test set, as suggested by Cremer et al. (2018) and Mattei & Frellsen (2018b).”