Operator Variational Inference

By Rajesh Ranganath, Dustin Tran, Jaan Altosaar, David M. Blei. 2016

121 citations recorded by the source corpus.

Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives. As one example, we design a variational objective with a Langevin-Stein operator. We develop a black box algorithm, operator variational inference (OPVI), for optimizing any operator objective. Importantly, operators enable us to make explicit the statistical and computational tradeoffs for variational inference. We can characterize different properties of variational objectives, such as objectives that admit data subsampling---allowing inference to scale to massive data---as well as objectives that admit variational programs---a rich class of posterior approximations that does not require a tractable density. We illustrate the benefits of OPVI on a mixture model and a generative model of images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, methodology): “The only other variational methods we are aware of that can deal with such expressive inference models are based on Stein Discrepancy (Ranganath et al., 2016; Liu & Feng, 2016).”
Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning (2016, methodology): “One exception is a very recent paper (Ranganath et al., 2016) that avoids calculating q ( x ) using an idea related to Stein discrepancy (Gorham & Mackey, 2015; Liu et al., 2016; Oates et al., 2014; Chwialkowski et al., 2016).”
Two Methods for Wild Variational Inference (2016, methodology): “…is related to a very recent work by Ranganath et al. (2016), which is based on directly minimizing the variational form of Stein discrepancy in (4); Ranganath et al. (2016) assumes F consists of a neural network φτ (z) parametrized by τ , and find η by solving the following min-max problem: min η…”
Wild Variational Approximations (2016, background): “Very recently [9] defined G as parametric functions represented by neural networks, and approximate the minimax optimisation with gradient descent (similar to GAN [45]).”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “An alternative Stein-type gradient estimator using the spectral decomposition of kernels was developed by Shi et al. (2018), and Stein’s identity was also used in variational inference by Ranganath et al. (2016).”
Kernel Implicit Variational Inference (2017, unknown): “…is called implicit distributions, and for variational methods that use an implicit variational posterior (also known as variational programs (Ranganath et al., 2016) or wild variational approximations (Liu & Feng, 2016)), we refer to them as Implicit Variational Inference (implicit VI).”
Variational Inference using Implicit Distributions (2017, background): “Operator variational in-Variational ference (OPVI, Ranganath et al., 2016) formulates a general class of variational lower bounds based on operator divergences, resulting in a practical algorithm for training implicit inference networks without a tractable density.”
Doubly Semi-Implicit Variational Inference (2018, methodology): “One example is operator variational inference [26] that uses the Langevin-Stein operator to design a new variational objective.”
Probabilistic programming with programmable inference (2018, methodology): “The interface Venture provides for inference tactics is also expressive enough in principle to support message-passing-based approaches to inference [22, 32] and to implement more flexible operator-based variational techniques [37].”
Deep Probabilistic Programming (2017, methodology): “We also use hierarchical variational models ( HVM s) (Ranganath et al., 2016b) with a normalizing ﬂow prior; it produced similar results as a normalizing ﬂow on the latent variable space (Rezende & Mohamed, 2015), and better than importance-weighted auto-encoders ( IWAE s) (Burda et al., 2016).”
Filtering Variational Objectives (2017, methodology): “[30] generalize the ELBO by considering arbitrary operators of the model and variational posterior.”
Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions (2017, methodology): “The proposed algorithm performs approximate inference by fitting an implicit distribution and therefore it shares similarities with other implicit variational inference methods such as [29, 22, 41, 18, 23, 16, 39].”
Continuous-Time Flows for Efficient Inference and Density Estimation (2017, methodology): “For density estimation on observed data, the generative adversarial network (GAN) can be regarded as an implicit density estimation method (Ranganath et al., 2016; Huszár, 2017; Mohamed & Lak-shminarayanan, 2017), in the sense that one may sample from the distribution (regarded as a representation…”
Simple, Distributed, and Accelerated Probabilistic Programming (2018, background): “, sample_shape=2) if eps[0] > 0: return neural_net_positive(eps[1], x) else: return neural_net_negative(eps[1], x) Figure 2: Variational program [35], available in eager mode.”
Gradient Estimators for Implicit Models (2017, background): “This is because many learning algorithms, e.g. maximum likelihood estimation (MLE), rely on minimising a distance/divergence/discrepancy measure D[ p || p D ] , which often requires evaluating the model density (c.f. [30, 22]).”