Importance Weighting and Variational Inference

By Justin Domke, D. Sheldon. 2018

120 citations recorded by the source corpus.

Recent work used importance sampling ideas for better variational bounds on likelihoods. We clarify the applicability of these ideas to pure probabilistic inference, by showing the resulting Importance Weighted Variational Inference (IWVI) technique is an instance of augmented variational inference, thus identifying the looseness in previous work. Experiments confirm IWVI's practicality for probabilistic inference. As a second contribution, we investigate inference with elliptical distributions, which improves accuracy in low dimensions, and convergence in high dimensions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Evaluating Lossy Compression Rates of Deep Generative Models (2020, neutral): “…p ( z )) in terms of the full KL divergence in the AIS extended state space: , where q AIS k ( z , V | x ) is constructed to have the marginal of q AIS k ( z | x ) (similar to the auxiliary variable construction of Domke & Sheldon (2018)), and p ( z ) p ( V | z , x ) has the marginal of p ( z ) .”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “1 under mild conditions [9]. as Bachman and Precup [3] introduced the idea of viewing IWAE auxiliary variable variational inference and Naesseth et al. [51], Cremer et al. [13], Domke and Sheldon [20] formalized the notion.”
MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets (2019, methodology): “Indeed, as studied by Cremer et al. (2017), Naesseth et al. (2018), and Domke & Sheldon (2018) the variational distribution qγ in (5) is replaced by a more complex distribution qIW which depends both on θ and γ.”
Importance Weighted Hierarchical Variational Inference (2019, background): “This bound has been shown (Domke and Sheldon, 2018) to be a tractable lower bound on ELBO for a variational distribution that has been obtained by self-normalizing importance sampling, or a special case of Sequential Monte Carlo (Maddison et al.”
Lifelong Generative Modelling Using Dynamic Expansion Graph Model (2021, methodology): “IWELBO bound (Burda, Grosse, and Salakhutdinov 2015) is an extension of ELBO by generating multiple weighted samples under the importance sampling (Domke and Sheldon 2018).”
AUTO-ENCODING VARIATIONAL BAYES (2020, unknown): “To evaluate the learned generative model, we provide goodness of fit metrics based on IWELBO and CUBO on held-out data, which are proxies for the log-likelihood.”
Deep Mixture Generative Autoencoders (2021, methodology): “Importance sampling [31], [32] is also used in VAEs for improving the quality of the inference.”