Variational Inference using Implicit Distributions

By Ferenc Huszár. 2017

145 citations recorded by the source corpus.

Generative adversarial networks (GANs) have given us a great tool to fit implicit generative models to data. Implicit distributions are ones we can sample from easily, and take derivatives of samples with respect to model parameters. These models are highly expressive and we argue they can prove just as useful for variational inference (VI) as they are for generative modelling. Several papers have proposed GAN-like algorithms for inference, however, connections to the theory of VI are not always well understood. This paper provides a unifying review of existing algorithms establishing connections between variational autoencoders, adversarially learned inference, operator VI, GAN-based image reconstruction, and more. Secondly, the paper provides a framework for building new algorithms: depending on the way the variational bound is expressed we introduce prior-contrastive and joint-contrastive methods, and show practical inference algorithms based on either density ratio estimation or denoising.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, methodology): “Some ideas of this paper were described independently by Huszár in a blog post on http://www. inference.vc and in Huszár (2017).”
Posterior Meta-Replay for Continual Learning (2021, background): “Since the expressivity of explicit distributions is limited, we also explore the more diverse variational family of implicit distributions (Goodfellow et al., 2014; Huszár, 2017).”
Variational Approaches for Auto-Encoding Generative Adversarial Networks (2017, background): “This trick is widespread [16, 18, 24, 25, 37], although perhaps not stated as explicitly.”
Distribution Matching in Variational Inference (2018, background): “The density ratio trick is a potential solution for the main challenges of VAEs: using implicit encoders and decoders (Huszár, 2017) can solve the lack of flexibility with the variational posterior and observation likelihoods and could fix the divergence minimization issues by directly matching marg”
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, methodology): “For example, we show that it can be used to provide accurate score estimates needed for variational inference with implicit distributions (Huszár, 2017) and learning Wasserstein Auto-Encoders (WAE, Tolstikhin et al.”
PixelGAN Autoencoders (2017, methodology): “Another type of GAN inference technique is used in the ALI [8] and BiGAN [9] models, which have been shown to approximate maximum likelihood learning [3].”
Generative Adversarial Networks (GANs) (2020, background): “The density ratio trick is widely used [4][119][72], [73], [133], [134][135] where class probability estimation is the most popular approach.”
Advances in Variational Inference (2017, background): “Implicit distributions for VI is an active area of research [72], [81], [102], [105], [107], [119], [129], [196], [210].”
Understanding Variational Inference in Function-Space (2020, background): “The second intractability can be addressed using any form of implicit inference (Husz´ar, 2017).”
Implicit Variational Inference with Kernel Density Ratio Fitting (2017, background): “Following the success of learning with implicit generative models, implicit distributions are applied to variational inference, which are surveyed in [10].”
Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference (2018, methodology): “, 2017), implicit variational models (Huszár, 2017), and adversarial message passing approximations (Karaletsos, 2016).”
Learning Latent Permutations with Gumbel-Sinkhorn Networks (2018, methodology): “These methods avoid evaluating the likelihood based on the observation that in many cases inference can be cast as the estimation of a likelihood ratio, which can be obtained from samples (Husz ´ ar, 2017).”
Learning Disentangled Representations with Reference-Based Variational Autoencoders (2019, background): “Interestingly, it has been shown that adversarial learning can be used to minimize the variational objective function of VAEs (Makhzani et al., 2016; Huszár, 2017).”
Automatic Posterior Transformation for Likelihood-Free Inference (2019, background): “Approaches for so-called implicit generative models generally require the simulator to be differentiable with respect to model parameters (Mohamed & Lakshminarayanan, 2016; Husz´ar, 2017, and references therein).”
Kernel Implicit Variational Inference (2017, methodology): “Most of the existing implicit VI methods (Mescheder et al., 2017; Husz´ar, 2017; Tran et al., 2017) rely on a discriminator to produce estimates of the variational objective and its gradients.”
Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation (2021, background): “On the contrary, a higher quality of qφ (zu,t |xu,[1:t ]) can result in a more flexible representational ability, as well as a better approximation to the true posterior [9].”
Doubly Semi-Implicit Variational Inference (2018, background): “This approach has not been validated in practice yet and has only been proposed as a theoretical concept [11].”
Adversarial Variational Optimization of Non-Differentiable Simulators (2017, background): “Likewise, AVO closely relates to recent extensions of GANs, such as Adversarial Learned Inference (ALI) [33], Bidirectional GANs (BiGANs) [34], α -GAN [31], Adversarial Variational Bayes (AVB) [35], and the PC-Adv al-gorithm of [36], which add an inference network to the generative model.”
Continuous-Time Flows for Efficient Inference and Density Estimation (2017, methodology): “…on observed data, the generative adversarial network (GAN) can be regarded as an implicit density estimation method (Ranganath et al., 2016; Huszár, 2017; Mohamed & Lak-shminarayanan, 2017), in the sense that one may sample from the distribution (regarded as a representation of the unknown…”