Automatic Differentiation Variational Inference

By Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, David M. Blei. 2016

827 citations recorded by the source corpus.

Probabilistic modeling is iterative. A scientist posits a simple model, fits it to her data, refines it according to her analysis, and repeats. However, fitting complex models to large data is a bottleneck in this process. Deriving algorithms for new models can be both mathematically and computationally challenging, which makes it difficult to efficiently cycle through the steps. To this end, we develop automatic differentiation variational inference (ADVI). Using our method, the scientist only provides a probabilistic model and a dataset, nothing else. ADVI automatically derives an efficient variational inference algorithm, freeing the scientist to refine and explore many models. ADVI supports a broad class of models-no conjugacy assumptions are required. We study ADVI across ten different models and apply it to a dataset with millions of observations. ADVI is integrated into Stan, a probabilistic programming system; it is available for immediate use.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Foundation Posteriors for Approximate Probabilistic Inference (2022, positive): “In this case, we propose to ﬁnetune our pretrained inference model using stochastic variational inference, or SVI [35, 50, 39].”
Variational Inference: A Review for Statisticians (2016, methodology): “…does not necessarily suffer in accuracy, for example, in terms of posterior predictive densities (Blei and Jordan 2006; Braun and McAuliffe 2010; Kucukelbir et al. 2017); other research focuses on where variational inference falls short, especially around the posterior variance, and tries to…”
SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, methodology): “In the stochastic optimization procedure, we adaptively set the step size as proposed in the automatic differentiation variational inference (advi) algorithm (Kucukelbir et al., 2017).”
The Variational Gaussian Process (2015, background): “In particular, the algorithm is linear in the number of latent variables, which is competitive with other variational inference methods.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “We compare Bayes by Backprop (BBB) (Blundell et al., 2015; Kucukelbir et al., 2017), which is a Bayesian neural network trained via gradient-based variational inference, Bayes by Backprop with NCP (BBB+NCP), and the OOD classifier model with NCP (Det+NCP).”
Hierarchical Variational Models (2015, methodology): “We introduce a distribution, r ( λ | z ; φ ) with parameters φ and apply the variational principle; As in variational inference, the bound in Eq.”
The Description Length of Deep Learning models (2018, background): “A common model class for β is the set of multivariate Gaussian distributions {N ( µ, Σ) , µ ∈ R d , Σ diagonal } , and µ and Σ can be optimized with a stochastic gradient descent algorithm (Graves, 2011; Kucukelbir et al., 2017).”
Analyzing the role of model uncertainty for electronic health records (2019, positive): “Model uncertainty has made many methodological advances in recent years—including reparameterization-based variational Bayesian neural networks [3, 7, 19, 21], Monte Carlo dropout [8], deep ensembles and efficient alternatives [20, 33], and function priors [9, 13, 22].”
The Generalized Reparameterization Gradient (2016, methodology): “In terms of accuracy, it outperforms automatic differentiation variational inference ( ADVI ) (Kucukelbir et al., 2016), which considers Gaussian variational distributions on a transformed space.”
Automated Scalable Bayesian Inference via Hilbert Coresets (2017, background): “…formulae are often onerous to obtain—and an approximating family—whose rigorous selection is an open question—ADVI (Ranganath et al., 2014; Kucukelbir et al., 2015, 2017) applies standard transformations to the model so that a multivariate Gaussian approximation can be used, and…”
Importance Weighting and Variational Inference (2018, background): “While Markov chain Monte Carlo is most commonly for these problems [9, 23], the high computational expense motivates VI [11, 3].”
Understanding Large-Scale Dynamic Purchase Behavior (2020, neutral): “Examples are stochastic subsampling, stochastic gradients, and parallelization of computations (Hoffman et al. 2013, Kucukelbir et al. 2017).”
Filtering Variational Objectives (2017, methodology): “jointly optimize the ELBO over p and q with stochastic gradient ascent [1, 2, 10, 11].”
A review of automatic differentiation and its efficient implementation (2018, methodology): “AD also plays a critical part in automatic differentiation variational inference (Kucukelbir et al., 2016), a gradient based method to approximate Bayesian posteriors.”
An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference? (2020, methodology): “Speciﬁcally, we ﬁt the microcredit model using a variant of Automatic Diﬀerentiation Variational Inference (ADVI) described in Giordano et al. (2018, Section 5.2) (see also the original ADVI paper, Kucukelbir et al. (2017)).”
Reweighted Data for Robust Probabilistic Models (2016, methodology): “Here we use ADVI , as NUTS struggles with mixture model inference (Kucukelbir et al., 2016).”