A Bayesian approach to on-line learning

By Manfred Opper. 1999

Subfields: Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Optimal Continual Learning has Perfect Memory and is NP-hard (2020, methodology): “For example, the work of Opper & Winther (1998) introduces an approximately Bayesian algorithm for neural networks that can be updated continuously in an on-line fashion.”
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting (2018, methodology): “Our work links the Kronecker factored Laplace approximation [34] to Bayesian online learning [31] similar to how Variational Continual Learning [30] connects Online Variational Learning [7, 13] to Bayes-by-Backprop [1].”
Assumed Density Filtering Methods for Learning Bayesian Neural Networks (2016, background): “…2014; Hern´andez-Lobato and Adams 2015), where the authors build upon the online Bayesian inference algorithm known as assumed density ﬁl-tering (ADF) (Opper 1998) to develop methods for learning Bayesian neural networks — expectation backpropagation (EBP) and probabilistic backpropagation (PBP).”
Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks (2015, methodology): “…method that approximates the exact posterior of a neural network (5) with a factored distribution given by The approximation parameters m ij,l , v ij,l , α γ , β γ , α λ and β λ are determined by applying an assumed density ﬁltering method (Opper & Winther, 1998; Minka, 2001) on the posterior (5).”
Expectation Propagation for approximate Bayesian inference (2001, background): “Figure 3 (b) plots cost vs. error for EP versus three other algorithms for estimating the Bayes point: the billiard al gorithm of Herbrich et al. (1999), the TAP algorithm of Opper & Winther (2000) , and the mean-field (MF) algo rithm of Opper & Winther (2000).”
Task Agnostic Continual Learning via Meta Learning (2019, methodology): “Bayesian Gradient Descent (BGD) (Zeno et al., 2018), as we discussed before, adopts the framework of online variational Bayes, and approximates the posterior with a diagonal Gaussian distribution.”
Measuring and regularizing networks in function space (2018, methodology): “…learning on an earlier task A, multiplied by the diagonal of the Fisher information matrix F (calculated on task A ): This idea is closely related to well-studied approaches to Bayesian online learning, if F is interpreted as a precision matrix (Honkela & Valpola (2003), Opper & Winther (1998)).”
Note on the quadratic penalties in elastic weight consolidation (2017, methodology): “EWC can be seen as an on-line approximate inference algorithm, closely related to assumed density ﬁltering [ADF Opper and Winther, 1998, Minka, 2001] and Laplace propagation [Eskin et al., 2004].”
Streaming Variational Bayes (2013, methodology): “This approach is similar in spirit to assumed density ﬁltering or expectation propagation [7–9], but each step of those methods involves a moment-matching step that can be computationally costly for models such as topic models.”
Statistical Mechanics of On-line Learning (2009, background): “These results for optimal modulations functions are, in general, better understood from a Bayesian point of view ([17,18,19]).”
Iterative Network Pruning with Uncertainty Regularization for Lifelong Sentiment Classification (2021, methodology): “Instead of keeping the old-tasks fixed when learning new tasks, we incorporate an uncertainty regularization [1, 6] based on the Bayesian online learning framework [27] into the iterative pruning procedure.”
Adversarial Phenomenon in the Eyes of Bayesian Deep Learning (2017, background): “Some of these works belong to two broad classes of approximate inference - variational inference (VI) and assumed density ﬁltering (ADF) [28] which we brieﬂy describe below.”
A family of algorithms for approximate Bayesian inference (2001, background): “ADF has been independently proposed in the statistics (Lauritzen, 1992; Bernardo & Giron, 1988; Stephens, 1997), artiicial intelligence (Boyen & Koller, 1998b; Opper & Winther, 1999; Barber & Sollich , 1999; Frey et al., 2000), and control literature (Kushner & Budhiraja, 2000; Maybeck, 1982).”
Bayesian Gaussian process models : PAC-Bayesian generalisation error bounds and sparse approximations (2003, methodology): “This simple idea has been used extensively, for example in the context of Bayesian on-line learning [138] or switching linear dynamical systems [10, 25, 127], see [125] for more exhaustive references.”
Pattern Recognition and Machine Learning (2006, unknown): “A special case of EP, known as assumed density filtering (ADF) or moment matching (Maybeck, 1982; Lauritzen, 1992; Boyen and Koller, 1998; Opper and Winther, 1999), is obtained by initializing all of the approximating factors except the first to unity and then making one pass through the factors upd”
Continual Learning with Bayesian Neural Networks for Non-Stationary Data (2020, methodology): “Prominent examples include sequential Monte Carlo (Liu & Chen, 1998), assumed density ﬁltering (Maybeck, 1982), and online variational Bayes (Opper, 1998; Ghahramani, 2000; Sato, 2001; Broderick et al., 2013).”
IncDet: In Defense of Elastic Weight Consolidation for Incremental Object Detection (2020, background): “elastic weight consolidation (EWC) based on the Bayesian online learning [16].”