Two Methods for Wild Variational Inference

By Qiang Liu, Yihao Feng. 2016

19 citations recorded by the source corpus.

Variational inference provides a powerful tool for approximate probabilistic inference on complex, structured models. Typical variational inference methods, however, require to use inference networks with computationally tractable probability density functions. This largely limits the design and implementation of variational inference methods. We consider wild variational inference methods that do not require tractable density functions on the inference networks, and hence can be applied in more challenging cases. As an example of application, we treat stochastic gradient Langevin dynamics (SGLD) as an inference network, and use our methods to automatically adjust the step sizes of SGLD to maximize its convergence speed, significantly outperforming the hand-designed step size schemes.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, methodology): “The only other variational methods we are aware of that can deal with such expressive inference models are based on Stein Discrepancy (Ranganath et al., 2016; Liu & Feng, 2016).”
Deep and Hierarchical Implicit Models (2017, background): “Further advances have analyzed variational programs (Ran-ganath et al., 2016a)—a family of approximations which only requires a process returning samples—and which has seen further interest (Liu & Feng, 2016).”
A Spectral Approach to Gradient Estimation for Implicit Distributions (2018, methodology): “It is based on a ridge regression that inverts a generalized version of Stein’s identity (Gorham & Mackey, 2015; Liu et al., 2016).”
Implicit Variational Inference with Kernel Density Ratio Fitting (2017, background): “This kind of distribution is called implicit distributions, and for variational methods that use an implicit variational posterior (also known as variational programs [25], wild variational approximations [16]), we refer to them as Implicit Variational Inference (Implicit VI).”
Variational Implicit Processes (2018, methodology): “In approximate inference context, implicit distributions have also been deployed to postulate flexible approximate posterior distributions [56, 58, 60, 61, 76, 87, 90].”
Stochastic Normalizing Flows (2020, methodology): “More recently, Liu & Feng (2016) considered optimizing step size in stochastic gradient Langevin dynamics using methods derived from kernelized Stein discrepancy.”
Hierarchical Implicit Models and Likelihood-Free Variational Inference (2017, background): “Further advances have analyzed variational programs [42]—a family of approximations which only requires a process returning samples—and which has seen further interest [30].”
Gradient Estimators for Implicit Models (2017, methodology): “This is because many learning algorithms, e.g. maximum likelihood estimation (MLE), rely on minimising a distance/divergence/discrepancy measure D[ p || p D ] , which often requires evaluating the model density (c.f. [30, 22]).”
Advances in Variational Inference (2017, background): “Implicit distributions for VI is an active area of research [72], [81], [102], [105], [107], [118], [129], [196], [210].”
Kernel Implicit Variational Inference (2017, methodology): “More recently, the amortized version of SVGD has been developed (Liu & Feng, 2016) and the same idea has been applied to MCMC (Li et al., 2017).”