Streaming Variational Bayes

By Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C. Wilson, Michael I. Jordan. 2013

368 citations recorded by the source corpus.

We present SDA-Bayes, a framework for (S)treaming, (D)istributed, (A)synchronous computation of a Bayesian posterior. The framework makes streaming updates to the estimated posterior according to a user-specified approximation batch primitive. We demonstrate the usefulness of our framework, with variational Bayes (VB) as the primitive, by fitting the latent Dirichlet allocation model to two large-scale document collections. We demonstrate the advantages of our algorithm over stochastic variational inference (SVI) by comparing the two after a single pass through a known amount of data—a case where SVI may be applied—and in the streaming setting, where SVI does not apply.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Decentralized Bayesian Learning over Graphs (2019, positive): “In particular, our learning rule also generalizes various Bayesian inference techniques such as [9–11, 5] and variational continual Notation : We use boldface for vectors v and denote its i -th element by v i .”
Variational Continual Learning (2017, methodology): “On the other hand, the online variational inference has been previously explored (Ghahramani and Attias, 2000; Broderick et al., 2013; Bui et al., 2017), but not for neural networks or in the context of complex tasks.”
Overcoming Catastrophic Forgetting by Incremental Moment Matching (2017, background): “Two algorithms have a different form of Gaussian with a different objective function for the same dataset.”
Continual Learning via Sequential Function-Space Variational Inference (2023, positive): “Variational continual learning ( VCL ; Nguyen et al., 2018; Swaroop et al., 2019), which extends the concept of online variational inference (Broderick et al., 2013; Ghahramani and Attias, 2000; Honkela and Valpola, 2003; Sato, 2001) to deep neural networks, uses the parameter covariance matrix of…”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, neutral): “[31] formulate the coreset summarization in continual learning using online variational inference [35, 6].”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, methodology): “…after observing data online, Bayes’ theorem provides a coherent framework to build continual learning algorithms, and avoid pitfalls like catastrophic forgetting; this principle is seen in Sato (2001) for online model selection, and in Broderick et al. (2013) for inference on streaming data.”
Optimal Continual Learning has Perfect Memory and is NP-hard (2020, background): “Similarly, the contributions to sequential and streaming learning made by Honkela & Valpola (2003), Broderick et al. (2013) or McInerney et al. (2015) are geared towards streaming settings in which the differences in observations a r X i v : 2006 .”
Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation (2019, background): “Variational continual learning (VCL) [27] combines the online variational inference [9, 31, 5] with Monte Carlo sampling [4] using neural networks, but it requires a relatively large amount of computational cost to infer an approximate posterior distribution.”
Lifelong Generative Modeling (2017, methodology): “Variational methods have been adapted for a streaming setting, e.g: Streaming Variational Bayes Broderick et al. (2013), Streaming Variational Mixture models Tank et al. (2015), and the Population Posterior McInerney et al. (2015).”
Task Agnostic Continual Learning Using Online Variational Bayes (2018, methodology): “In online variational Bayes (Broderick et al., 2013), we aim to find the posterior in an online setting, where the data arrives sequentially.”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “Prior algorithms have used online gradient updates (Duchi et al., 2011) and probabilistic ﬁltering formulations (Murphy, 2002; Hoffman et al., 2010; Broderick et al., 2013).”
Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates (2020, background): “In online variational Bayes (Broderick et al., 2013), one aims to find the posterior in an online setting, where data arrive sequentially.”
Streaming Sparse Gaussian Process Approximations (2017, methodology): “The approach also relates to the streaming variational Bayes framework [10].”
Coresets for Scalable Bayesian Logistic Regression (2016, methodology): “and streaming variational Bayes [6, 7] lead to gains in computational efficiency, but lack rigorous justification and provide no guarantees on the quality of inference.”
Automated Scalable Bayesian Inference via Hilbert Coresets (2017, methodology): “Examples include subsampling and streaming methods for variational Bayes (Hoﬀman et al., 2013; Broderick et al., 2013; Campbell et al., 2015), subsampling methods for MCMC (Welling and Teh, 2011; Ahn et al., 2012; Bardenet et al., 2014; Korattikara et al., 2014; Maclaurin and Adams, 2014; Bardenet…”
Edward: A library for probabilistic modeling, inference, and criticism (2016, methodology): “The approach of Bayesian filtering (Doucet et al., 2000; Broderick et al., 2013) can be applied in Edward using recursive posterior inferences; the approach of population posteriors (McInerney et al.”
Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes (2020, methodology): “In contrast, streaming variational inference [32, 33, 34] for Bayesian nonparametric models is designed for handling streaming data and requires a single computation pass.”
A Stochastic Variational Framework for Fitting and Diagnosing Generalized Linear Mixed Models (2012, methodology): “ptimization before switching to the standard version. Recently, there is increasing interest in developing VB algorithms capable of handling large data sets or streaming data (e.g. Luts et al., 2013; Broderick et al., 2013). Stochastic optimization is an important tool in parameter estimation for la”
Streaming Variational Inference for Bayesian Nonparametric Mixture Models (2014, methodology): “In contrast, streaming variational Bayes (SVB) [2] handles unbounded data sets by exploiting the sequential nature of Bayes theorem to recursively update an approximation of the posterior.”
Continual Learning with Bayesian Neural Networks for Non-Stationary Data (2020, methodology): “Prominent examples include sequential Monte Carlo (Liu & Chen, 1998), assumed density ﬁltering (Maybeck, 1982), and online variational Bayes (Opper, 1998; Ghahramani, 2000; Sato, 2001; Broderick et al., 2013).”