Automatic Variational Inference in Stan

By Alp Kucukelbir, Rajesh Ranganath, Andrew Gelman, David M. Blei. 2015

243 citations recorded by the source corpus.

Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult for non-experts to use. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI); we implement it in Stan (code available), a probabilistic programming system. In ADVI the user provides a Bayesian model and a dataset, nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, methodology): “We compare AVB against two variational methods with Gaussian inference model (Kucukelbir et al., 2015) as implemented in STAN (Stan Development Team, 2016).”
Variational Inference: A Review for Statisticians (2016, methodology): “They focused on a broader class of models (with posterior support in real coordinate space) and analyzed an automated VI technique that uses a Gaussian variational approximation (Kucukelbir et al. 2015).”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, methodology): “Skin Protein constant SGD 18.7 0.471 1000.9 constant SGD-d 14.0 0.921 678.4 constant SGD-f 0.7 0.005 1.8 SGLD (Welling and Teh, 2011) 2.9 0.905 4.5 SGFS-d (Ahn et al., 2012) 12.8 0.864 597.4 SGFS-f (Ahn et al., 2012) 0.8 0.005 1.3 BBVI (Kucukelbir et al., 2015) 44.7 5.74 478.1 Table 1: KL divergence”
Hierarchical Variational Models (2015, background): “Lawrence (2000) explores a rich class of variational approximations formed by both mixtures and Markov chains.”
Automatic Differentiation Variational Inference (2016, methodology): “1This paper extends the method presented in (Kucukelbir et al., 2015).”
The Generalized Reparameterization Gradient (2016, methodology): “We apply the sparse gamma DEF on two different databases: (i) the Olivetti database at AT&T, 6 which consists of 400 (320 for training and 80 for test) 64 × 64 images of human faces in a 8 bit scale ( 0 − 255 ); and (ii) the collection of papers at the Neural Information Processing Systems ( NIPS ) ”
Automated Scalable Bayesian Inference via Hilbert Coresets (2017, background): “…formulae are often onerous to obtain—and an approximating family—whose rigorous selection is an open question—ADVI (Ranganath et al., 2014; Kucukelbir et al., 2015, 2017) applies standard transformations to the model so that a multivariate Gaussian approximation can be used, and computes…”
Black-box $\alpha$-divergence Minimization (2015, background): “[Kucukelbir et al., 2015, Ranganath et al., 2014, Salimans et al., 2013].”
A Variational Analysis of Stochastic Gradient Algorithms (2016, methodology): “Our results show that the approximation to the posterior given by constant SGD is not worse than the approximation given by BBVI.”
Bayesian Incremental Learning for Deep Neural Networks (2018, methodology): “Gaussian approximation is widely used (Blundell et al., 2015; Kingma et al., 2015; Kucukelbir et al., 2015; Molchanov et al., 2017), however this family has low expressiveness (Louizos & Welling, 2017) which affects the quality of incremental learning.”
Denotational validation of higher-order Bayesian inference (2017, methodology): “Most of these research efforts have focussed on developing efficient inference algorithms and implementations [Kucukelbir et al. 2015; Le et al. 2017; Tran et al. 2017; Wingate andWeber 2013].”
Black-Box Policy Search with Probabilistic Programs (2015, methodology): “This is in contrast to implementations based on (reverse-mode) automatic differentiation [Pearlmutter and Siskind, 2008], as is used in Stan [Kucukelbir et al., 2015], which store derivative terms for the entire computation graph.”
The Virtual Epileptic Patient: Individualized whole‐brain models of epilepsy spread (2017, methodology): “This approach takes advantage of the ef ﬁ ciency of the variational algorithm, which constructs an approximate proxy distribution on the true posterior optimized via stochastic gradient ascent (Kucukelbir et al., 2015).”
Boosting Variational Inference (2016, methodology): “For comparison, we run automatic differentiation variational inference (ADVI) [Kucukelbir et al., 2015], which effectively performs mean-field variational inference (MFVI) but does not require any conditional conjugacy properties in the generative model.”
Rényi Divergence Variational Inference (2016, methodology): “The resulting method can be applied to any model that MC-VI [12, 13, 14, 15] is applied to.”
Reweighted Data for Robust Probabilistic Models (2016, methodology): “For this model, we use ADVI .”
Deep Amortized Inference for Probabilistic Programs (2016, background): “Continuous random variables which are parameterized by a location and a scale parameter naturally support this type of transformation, and other types of continuous variables can often be well-approximated by deterministic transformations of unit normal variables [17].”