Doubly Semi-Implicit Variational Inference

By Dmitry Molchanov, Valery Kharitonov, Artem Sobolev, Dmitry P. Vetrov. 2018

42 citations recorded by the source corpus.

We extend the existing framework of semi-implicit variational inference (SIVI) and introduce doubly semi-implicit variational inference (DSIVI), a way to perform variational inference and learning when both the approximate posterior and the prior distribution are semi-implicit. In other words, DSIVI performs inference in models where the prior and the posterior can be expressed as an intractable infinite mixture of some analytic density with a highly flexible implicit mixing distribution. We provide a sandwich bound on the evidence lower bound (ELBO) objective that can be made arbitrarily tight. Unlike discriminator-based and kernel-based approaches to implicit variational inference, DSIVI optimizes a proper lower bound on ELBO that is asymptotically exact. We evaluate DSIVI on a set of problems that benefit from implicit priors. In particular, we show that DSIVI gives rise to a simple modification of VampPrior, the current state-of-the-art prior for variational autoencoders, which improves its performance.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “Our second contribution is to show that EIMs provide a unifying conceptual framework to explain many advances in constructing tighter variational lower bounds for latent variable models ( e.g. , [9, 48, 51, 41, 68, 50, 63]).”
Importance Weighted Hierarchical Variational Inference (2019, background): “, 2016), Semi-Implicit Variational Inference (Yin and Zhou, 2018) and Doubly Semi-Implicit Variational Inference (Molchanov et al., 2018).”
Generalized ELBO with Constrained Optimization , GECO (2018, background): “As a result, several mechanisms have been proposed to increase the expressiveness of the variational posterior density [6, 7, 8, 9, 10, 11, 12, 13, 14] but it still remains a challenge to train complex encoders due to a combination of optimization and generalization issues [15, 16].”
VariGrow: Variational Architecture Growing for Task-Agnostic Continual Learning based on Bayesian Novelty (2022, methodology): “Meanwhile, each component is handled by an expert distribution defined implicitly through Bayesian Neural Networks (BNNs) (Yin & Zhou, 2018; Titsias & Ruiz, 2019; Molchanov et al., 2019).”
Lifelong Generative Modelling Using Dynamic Expansion Graph Model (2021, background): “2016), importance sampling (Burda, Grosse, and Salakhutdinov 2015; Domke and Sheldon 2018) or through hierarchical variational models (Molchanov et al. 2019; Vahdat and Kautz 2020), has been successful for improving the performance of VAEs.”
Learning Hierarchical Priors in VAEs (2019, methodology): “staurant Process prior [13], Gaussian mixture priors [12], or autoregressive priors [10]. A closely related line of research is based on the insight that the optimal prior is the aggregated posterior [28, 19]. The VampPrior [28] approximates the prior by a uniform mixture of approximate posterior di”