On-line learning and stochastic approximations

By L. Eon Bottou. 1999

Subfields: Optimization & Training, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Statistical Learning Theory

Citations in the corpus

Byzantine-Resilient Secure Federated Learning (2020, background): “The two conditions ∑∞ t=1 γ (t) = ∞ and ∑t=1(γ(t))2 < ∞ are instrumental in the convergence of stochastic gradient descent algorithms [37].”
GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning (2021, neutral): “In the offline setting, each new task’s data is available in entirety for repeated/ iterative learning [42], whereas online CL only considers data availability in small buffers that cannot be revisited [43].”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “+ on-line learning + forward transfer – no backward transfer – no knowledge retention – single task/domain (Bottou, 1999) (Bottou and LeCun, 2004) (Cesa-Bianchi and Lugosi, 2006) (Shalev-Shwartz, 2012) (C.”
Task-Free Continual Learning (2018, neutral): “We refer to [5, 33] for surveys and overviews on the topic.”
Lifelong Generative Modeling (2017, background): “The idea of learning in a continual manner has been explored extensively in machine learning, seeded by the seminal works of lifelong-learning [126, 125, 117], online-learning [32, 9, 11, 12] and sequential linear gaussian models [105, 36] such as the Kalman Filter [56] and its non-linear counterpar”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, background): “ converged. Note that their notion of hyperparameters does not refer to model parameters but to parameters of SGD. Stochastic Optimization. Stochastic gradient descent is an active ﬁeld (Zhang, 2004; Bottou, 1998). Many papers discuss constant step-size SGD. Bach and Moulines (2013); Flammarion and ”
Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging (2014, background): “[7] Léon Bottou, “Online learning and stochastic approximations,” On-line learning in neural networks, vol.”
Automatic differentiation in machine learning: a survey (2015, methodology): “onsiderable eﬀort on the manual derivation of analytical derivatives to subsequently plug these into standard optimization procedures such as L-BFGS (Zhu et al., 1997) or stochastic gradient descent (Bottou, 1998). Manual diﬀerentiation is time consuming and prone to error. Of the other alternatives”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “In principle, commonly used gradient-based learning methods, such as SGD, can easily be used as online learning algorithms (Bottou, 1998).”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “Several works have elaborated on challenges in: online learning in NNs (Heskes and Kappen, 1993), more generally online learning and stochastic approximations (Bottou, 1999; Saad, 1999), or speciﬁcally online gradient descent (Zinkevich, 2003), the workhorse of modern optimization.”
Making AI Forget You: Data Deletion in Machine Learning (2019, methodology): “Data Deletion as an Online Problem One convenient way of concretely formulating the computational challenge of data deletion is via the lens of online algorithms [17].”
Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks (2015, methodology): “We perform optimization using stochastic gradient descent (Bottou, 1998).”
Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement (2016, background): “propagation (Bottou, 1998), with a learning rate fixed to 10−4.”
Learning Deep Representations for Graph Clustering (2014, methodology): “In addition, whereas EVD is hard to parallel, the stochastic gradient descent (SGD) (Bottou 1998) training of DNN is comparatively easy to parallel, as have been well explored in the literature of DNN.”
A Variational Analysis of Stochastic Gradient Algorithms (2016, background): “Stochastic gradient descent is an active field (Zhang, 2004; Bottou, 1998).”
Identifying suspicious URLs: an application of large-scale online learning (2009, background): “SGD has received renewed attention because of recent results on the convergence of SGD algorithms and the casting of classic algorithms as SGD approximations (Bottou, 1998; Bottou & LeCun, 2004).”
The Population Posterior and Bayesian Modeling on Streams (2015, methodology): “Stochastic optimization maximizes an objective by following noisy and unbiased gradients [7, 19].”