Structured Stochastic Variational Inference

By Matthew D. Hoffman, David M. Blei. 2014

97 citations recorded by the source corpus.

Stochastic variational inference makes it possible to approximate posterior distributions induced by large datasets quickly using stochastic optimization. The algorithm relies on the use of fully factorized variational distributions. However, this "mean-field" independence approximation limits the fidelity of the posterior approximation, and introduces local optima. We show how to relax the mean-field approximation to allow arbitrary dependencies between global parameters and local hidden variables, producing better parameter estimates by reducing bias, sensitivity to local optima, and sensitivity to hyperparameters.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, methodology): “Hoffman, Blei, and Mimno (2012) and Hoffman and Blei (2015) studied MCMC as a method of approximating coordinate updates, for example, to include structure in the variational family.”
Coordinated Multi-Agent Imitation Learning (2017, background): “We focus on the structured mean-ﬁeld variational family, which factor-izes q as q p z, θ q “ q p z q q p θ q (Hoffman & Blei, 2014) and decomposes the ELBO objective: L “ E q r log p p θ s ´ E q r log q p θ s ` E q r log p p p z, x | θ qs ´ E q r log p q p z qqs .”
Advances in Variational Inference (2017, background): “For example, structured variational inference for LDA [66] shows that maintaining global structure is vital, while structured variational inference for the Beta Bernoulli Process [175] shows that maintaining local structure is more important.”
Hierarchical Indian buffet neural networks for Bayesian continual learning (2019, methodology): “We derive a structured variational posterior where dependencies are established between global parameters and local parameters [Hoffman and Blei, 2015].”
The Population Posterior and Bayesian Modeling on Streams (2015, methodology): “In classical VB, where we approximate the usual posterior, we cannot compute the KL.”
Doubly Semi-Implicit Variational Inference (2018, background): “They can be divided roughly into two main groups: those that require the probability density function of the approximate posterior to be analytically tractable (which we will call explicit models) [30, 12, 9, 5, 36, 38, 6, 27, 20] and those that do not (implicit models) [11, 22, 37, 17, 21, 31, 40].”
Adversarial Message Passing For Graphical Models (2016, background): “These factorizations have been explained at length in the context of stochastic inversion [11] and form a structured inverse factorization as used in variational inference [12], while also being widely used in the message passing literature [13],[14].”
Variational Beam Search for Novelty Detection (2021, background): “We ﬁrst deﬁne the marginal likelihood over data given the change variable s t to be p β ( x t | s t ) := (cid:82) p ( x t | z t ) p β ( z t | s t ) d z t , which is often intractable. posterior , x ) This intractability leaves the exact over s t p β ( s t | t not available as well (by Bayes rule). i”
Deep Probabilistic Programming (2017, background): “…one update of each (Neal & Hinton, 1993), This extends to many other cases such as exact EM for exponential families, contrastive divergence (Hinton, 2002), pseudo-marginal methods (Andrieu & Roberts, 2009), and Gibbs sampling within variational inference (Wang & Blei, 2012; Hoffman & Blei, 2015).”
Indian Buffet Neural Networks for Continual Learning (2019, methodology): “Because the mean-ﬁeld approximation removes much of the structure of a hierarchical model like the IBP, [20] uses structured stochastic variational inference [15] to allow dependencies between global and local parameters and achieve better results in VAEs over the mean-ﬁeld approximation.”