Online Model Selection Based on the Variational Bayes

By Masa-aki Sato. 2001

432 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, background): “For more on natural gradients and variational inference, see Sato (2001) and Honkela et al. (2008).”
Variational Continual Learning (2017, methodology): “…for the four projection operators previously identiﬁed, to i) Laplace propagation (Smola et al., 2004), ii) online VI (Ghahramani and Attias, 2000; Sato, 2001) also known as streaming VB (Broderick et al., 2013), iii) assumed density ﬁltering (Maybeck, 1982) and iv) sequential Monte Carlo (Liu…”
Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “…et al., 2018; Swaroop et al., 2019), which extends the concept of online variational inference (Broderick et al., 2013; Ghahramani and Attias, 2000; Honkela and Valpola, 2003; Sato, 2001) to deep neural networks, uses the parameter covariance matrix of the model currently serving as the prior.”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, positive): “[31] formulate the coreset summarization in continual learning using online variational inference [35, 6].”
Variational Auto-Regressive Gaussian Processes for Continual Learning (2020, methodology): “The recursive application of Bayes’ theorem provides us with a principled approach to update beliefs as new data arrives [30, 2] and much literature has been devoted to scalable Bayesian updating schemes – regularization using a Laplace approximation and diagonal Fisher information matrix [15] or ap”
Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation (2019, background): “Variational continual learning (VCL) [27] combines the online variational inference [9, 31, 5] with Monte Carlo sampling [4] using neural networks, but it requires a relatively large amount of computational cost to infer an approximate posterior distribution.”
Streaming Sparse Gaussian Process Approximations (2017, methodology): “For fixed hyperparameters and pseudo-points, the new VFE framework is equivalent to the application of streaming variational Bayes (VB) or online variational inference [10, 17, 18] to the GP setting in which the previous posterior plays a role of an effective prior for the new data.”
How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning (2019, methodology): “However the posterior distribution p ( θ | d 1: T ) is intractable, hence we use q t ( θ ) to approximate it through a KL divergence minimization (Sato, 2001), for t ≥ 1 .”
Dirichlet Process Mixtures of Generalized Linear Models (2009, methodology): “esults were generated using MCMC, which does not scale well to large datasets. An alternative implementation using variational inference (Blei and Jordan 2005), possibly online variational inference (Sato 2001), would greatly increase computational feasibility for large datasets. Our empirical analy”
Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions (2015, methodology): “n community, where it is known as mirror descent [Beck and Teboulle, 2003]. However, there are only a small number of variational methods that exploit the geometry of the problem in such a way (e.g. [Sato, 2001, Honkela et al., 2011]). A third and ﬁnal issue is about the structure of the lower bound”
Variational algorithms for approximate Bayesian inference (2003, methodology): “I briefly present here an online version of the VBEM algorithm above (but see also Ghahramani and Attias , 2000; Sato, 2001).”
Semiparametric Mean Field Variational Bayes: General Principles and Numerical Issues (2016, background): “Exponential family distributions also play an important role in the theory of mean field variational Bayes (e.g. Sato, 2001; Beal and Ghahramani, 2006; Wainwright and Jordan, 2008).”
Incremental learning of nonparametric Bayesian mixture models (2008, background): “Sato [14] derives recursive update rules for Variational Bayesian learning of mixture models.”
On-line Variational Bayesian Learning (2003, methodology): “While revising the paper we became aware that essentially the same method has been proposed earlier in [18].”
Recursive Noise Adaptive Kalman Filtering by Variational Bayesian Approximations (2009, methodology): “The fixedpoint algorithm is in fact a type of natural gradient method, which can be shown to have asymptotic optimal convergence properties [25].”
One-class collaborative filtering with random graphs (2013, background): “We remark, too, that the natural parameters in (15) define the natural gradients of the variational objective function [1, 24].”
Continual Learning with Bayesian Neural Networks for Non-Stationary Data (2020, methodology): “Prominent examples include sequential Monte Carlo (Liu & Chen, 1998), assumed density ﬁltering (Maybeck, 1982), and online variational Bayes (Opper, 1998; Ghahramani, 2000; Sato, 2001; Broderick et al., 2013).”
Kullback-Leibler Proximal Variational Inference (2015, methodology): “Relationship to the methods that use natural gradients: An alternative approach to incorporate the geometry of the posterior distribution is to use natural gradients [6, 5, 1].”
Natural Conjugate Gradient in Variational Inference (2007, methodology): “Both VB-E and VBM steps can implicitly optimally utilize the Riemannian structure of q(θ|ξ) for conjugate exponential family models [10].”