On-Line Learning in Neural Networks

By D. Saad. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical Learning Theory

Citations in the corpus

Continual Learning in the Teacher-Student Setup: Impact of Task Similarity (2021, background): “…series of classic papers by Biehl & Schwarze (1995) and Saad & Solla (1995a) who derived a set of closed ODEs that track the test error of the student (see also Saad & Solla (1995b); Biehl et al. (1996); Saad (2009) for further results and Goldt et al. (2019) for a recent proof of these equations).”
Assumed Density Filtering Methods for Learning Bayesian Neural Networks (2016, methodology): “…2014; Hern´andez-Lobato and Adams 2015), where the authors build upon the online Bayesian inference algorithm known as assumed density ﬁl-tering (ADF) (Opper 1998) to develop methods for learning Bayesian neural networks — expectation backpropagation (EBP) and probabilistic backpropagation (PBP).”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “f frameworks for empirical risk optimization [58], [59]. Several works have elaborated on challenges in online learning in NNs [60], more generally online learning and stochastic approximations [61], [62] or speciﬁcally online gradient descent [63], the workhorse of modern optimization. Given the in”
Gaussian processes:iterative sparse approximations (2002, methodology): “[1999] and Schölkopf et al. [1999]) using a batch framework and maximum a-posteriori solutions.”
Understanding Short-Horizon Bias in Stochastic Meta-Optimization (2018, background): “Learning rate decay is also required to achieve convergence guarantee for stochastic gradient methods under certain conditions (Bottou, 1998).”
Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning (2001, methodology): “These are methods and systems for: adaptive learning [4,5,7,8,14,30,46,47,48]; incremental learning [6,7,8,9,19,53,58,61,71]; lifelong learning [69,35,36,82]; on-line learning [17,21,22,28,31,35,36,42,44,61,66,67,69]; constructivist structural learning [15,19,11,14,9] that is supported by biological”
The Loss Surfaces of Multilayer Networks (2014, methodology): “In the 1990s, a number of researchers studied the convergence of gradient-based learning for multilayer networks using the methods of statistical physics, i.e. [Saad and Solla, 1995], and the edited works [Saad, 2009].”
Adaptive natural gradient learning algorithms for various stochastic models (2000, methodology): “Moreover, it has theoretically rigorous justification (Rattray and Saad, 1999).”
Machine learning: a review of classification and combining techniques (2006, methodology): “In detail, the idea of rule extraction from a neural network involves certain procedures, specifically the reading of parameters from a network, which is not allowed by the traditional connectionist framework that these neural networks are based on. Neural networks are usually more able to easily pr”