Stochastic variational inference
By Matthew D. Hoffman, David M. Blei, Chong Wang, John William Paisley. 2012
2,851 citations recorded by the source corpus.
We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Using stochastic variational inference, we analyze several large collections of documents: 300K articles from Nature, 1.8M articles from The New York Times, and 3.8M articles from Wikipedia. Stochastic inference can easily handle data sets of this size and outperforms traditional variational inference, which can only handle a smaller subset. (We also show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Stochastic variational inference lets us apply complex Bayesian models to massive data sets.
Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Ruslan Salakhutdinov
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Kyunghyun Cho
In Statistics and Probability
- David M. Blei
- Qiang Liu
- Michalis K. Titsias
- Rajesh Ranganath
- Zoubin Ghahramani
- Stephan Mandt
- Matthew D. Hoffman
- Richard E. Turner
In Probabilistic & Bayesian Methods
- David M. Blei
- Zoubin Ghahramani
- Richard E. Turner
- Max Welling
- Stefano Ermon
- Qiang Liu
- Rajesh Ranganath
- Yarin Gal
Citations in the corpus
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “The introduction of stochastic backpropagation (Paisley et al., 2012; Hoffman et al., 2013) and the variational auto-encoder (VAE) (Kingma, 2013; Rezende et al.”
- Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “Stochastic variational inference Blei et al. (2012); Hoffman et al. (2013) is a method for scalable posterior inference with large datasets using stochastic gradient ascent.”
- Density estimation using Real NVP (2016, background): “Its success in leveraging recent advances in backpropagation [51, 39] in deep neural networks resulted in its adoption for several applications ranging from speech synthesis [12] to language modeling [8].”
- Variational Inference with Normalizing Flows (2015, background): “Variational inference now lies at the core of large-scale topic models of text (Hoffman et al., 2013), provides the state-of-the-art in semi-supervised classification (Kingma et al., 2014), drives the models that currently produce the most realistic generative models of images (Gre-gor et al., 2014;…”
- Foundation Posteriors for Approximate Probabilistic Inference (2022, positive): “In this case, we propose to finetune our pretrained inference model using stochastic variational inference, or SVI [35, 50, 39].”
- Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro (2019, methodology): “As a final example, in Stochastic Variational Inference (SVI) [17], we optimize a loss function that is a Monte Carlo estimate of the Evidence Lower Bound (ELBO).”
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning (2015, methodology): “Recent advances in variational inference introduced new techniques into the field such as sampling-based variational inference and stochastic variational inference (Blei et al., 2012; Kingma & Welling, 2013; Rezende et al., 2014; Titsias & L´azaro-Gredilla, 2014; Hoffman et al., 2013).”
- Optimization Methods for Large-Scale Machine Learning (2016, methodology): “e the discovery of the natural gradient algorithm is closely associated with information geometry, nearly all its applications involve density estimation [5, 33] or conditional probability estimation [119,76,99] using objective functions that are closely connected to the KL divergence. Natural gradi”
- Black Box Variational Inference (2013, methodology): “Second, we develop generic stochastic variational inference (Hoffman et al., 2013), where we additionally subsample from the data to more cheaply compute noisy gradients.”
- On the challenges of learning with inference networks on sparse, high-dimensional data (2017, methodology): “In contrast, older stochastic variational inference methods [Hoffman et al., 2013] update θ based on gradients of L(x; θ, ψ∗) by updating randomly initialized variational parameters for each example.”
- Composing graphical models with neural networks for structured representations and fast inference (2016, methodology): “In contrast to the approaches of Khan et al. [23], Khan et al. [24], and Knowles et al. [25], rather than linearizing intractable terms around the current iterate, in this work we handle intractable terms via recognition networks and amoritized inference (and the remaining tractable objective terms ”
- SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, background): “In language, vector representations of words (also called “distributed representations”) allow us to reason about their usage and meaning (Harris, 1954; Firth, 1957; Bengio et al., 2003; Mikolov et al., 2013a).”
- Deep Gaussian Processes (2012, background): “A very promising approach would be to apply stochastic variational inference [Hoffman et al., 2012].”
- Probabilistic Model-Agnostic Meta-Learning (2018, background): “Although exact inference in Bayesian neural networks is impractical, approximations based on backpropagation and sampling [16, 32, 20, 2] have been effective in incorporating uncertainty into the weights of generic networks.”
- GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, methodology): “(Hoffman et al., 2013) developed a stochastic optimization process for VI.”
- Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, methodology): “By utilizing scalable sparse approximations for Gaussian processes (Titsias, 2009; Hensman et al., 2013; 2015) and advances in variational inference (Hoffman et al., 2013), we make the following key contributions — i) a generalized continual variational lower bound for sequential datasets with a…”