On-line Variational Bayesian Learning

By Antti Honkela, Harri Valpola. 2003

91 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Continual Learning (2017, methodology): “We merge online variational inference (VI) (Ghahramani and Attias, 2000; Sato, 2001; Broderick et al., 2013) with Monte Carlo VI for neural networks (Blundell et al.”
Overcoming Catastrophic Forgetting by Incremental Moment Matching (2017, background): “Two algorithms have a different form of Gaussian with a different objective function for the same dataset.”
Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “…learning ( VCL ; Nguyen et al., 2018; Swaroop et al., 2019), which extends the concept of online variational inference (Broderick et al., 2013; Ghahramani and Attias, 2000; Honkela and Valpola, 2003; Sato, 2001) to deep neural networks, uses the parameter covariance matrix of the model…”
Optimal Continual Learning has Perfect Memory and is NP-hard (2020, background): “Similarly, the contributions to sequential and streaming learning made by Honkela & Valpola (2003), Broderick et al. (2013) or McInerney et al. (2015) are geared towards streaming settings in which the differences in observations a r X i v : 2006 .”
Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation (2019, background): “Variational continual learning (VCL) [27] combines the online variational inference [9, 31, 5] with Monte Carlo sampling [4] using neural networks, but it requires a relatively large amount of computational cost to infer an approximate posterior distribution.”
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting (2018, methodology): “This type of Bayesian online learning has been studied extensively in the literature [31, 7, 13].”
Lifelong Generative Modeling (2017, methodology): “Past work in sequential learning of generative models has focused on learning Gaussian mixture models Singer & Warmuth (1999); Declercq & Piater (2008) or on variational methods such as Variational EM Ghahramani & Attias (2000).”
Measuring and regularizing networks in function space (2018, methodology): “…learning on an earlier task A, multiplied by the diagonal of the Fisher information matrix F (calculated on task A ): This idea is closely related to well-studied approaches to Bayesian online learning, if F is interpreted as a precision matrix (Honkela & Valpola (2003), Opper & Winther (1998)).”
Streaming Variational Bayes (2013, methodology): “We have demonstrated the advantages of our algorithm over stochastic variational inference and the sufﬁcient statistics update algorithm, particularly with respect to the key issue of obtaining approximations to posterior probabilities based on the number of documents seen thus far, not posterior…”
Streaming Sparse Gaussian Process Approximations (2017, methodology): “For fixed hyperparameters and pseudo-points, the new VFE framework is equivalent to the application of streaming variational Bayes (VB) or online variational inference [10, 17, 18] to the GP setting in which the previous posterior plays a role of an effective prior for the new data.”
Uncertainty in Deep Learning (2016, methodology): “the approximating 1Although online VI methods processing one point at a time have a long history [Ghahramani and Attias, 2000; Sato, 2001]”
Continual Learning with Bayesian Neural Networks for Non-Stationary Data (2020, methodology): “Prominent examples include sequential Monte Carlo (Liu & Chen, 1998), assumed density ﬁltering (Maybeck, 1982), and online variational Bayes (Opper, 1998; Ghahramani, 2000; Sato, 2001; Broderick et al., 2013).”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, methodology): “More specifically we unify work on: online VI [Ghahramani and Attias, 2000, Sato, 2001, Broderick et al., 2013, Bui et al., 2017b, Nguyen et al., 2018]; global VI [Sato, 2001, Hensman et al., 2012, Hoffman et al., 2013, Salimans and Knowles, 2013, Sheth and Khardon, 2016a, Sheth et al., 2015, Sheth…”
Advances in Variational Inference (2017, methodology): “Stochastic Variational Inference amounts to applying stochastic optimization to the objective function encountered in VI [65], [67], [71], [209], thereby scaling VI to very large datasets.”
Sentiment Classification by Leveraging the Shared Knowledge from a Sequence of Domains (2019, background): “Several methods, under the name of continual learning, have been proposed to overcome catastrophic forgetting, e.g., system-level consolidation [15,16] and sequential Bayesian [2,11].”
Regression Density Estimation With Variational Methods and Stochastic Approximation (2012, methodology): “These mixtures of heteroscedastic regression (MHR) models extend conventional mixtures of regression models (Jacobs et al. 1991; Jordan and Jacobs 1994) by allowing the component models to be heteroscedastic. Mixtures of regression models are usually referred to as mixtures of experts models in mach”
Dynamic Language Models for Streaming Text (2014, background): “There has also been work on online and stochastic learning for Bayesian models (Sato, 2001; Honkela and Valpola, 2003; Hoffman et al., 2013), based on variational inference.”