Online Learning and Stochastic Approximations

By L. Eon Bottou. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

A Closer Look at Memorization in Deep Networks (2017, background): “Because DNN optimization is not well-understood, it is unclear which of these hypotheses can actually be reached by gradient-based training (Bottou, 1998).”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, methodology): “In (Li et al., 2014), the authors argue that the convergence rate of SGD for the large-batch setting can be improved by obtaining iterates through the following proximal sub-problem. x k +1 = arg min (5) The motivation for this strategy is, in the context of large-batch methods, to better utilize a ”
Byzantine-Resilient Secure Federated Learning (2020, background): “lows [40] and [9], where we prove the convergence of the”
The Hidden Vulnerability of Distributed Learning in Byzantium (2018, background): “Q , depending on a parameter x , if one keeps updating x in the opposite direction of the gradient of Q , with reasonably small, but not too small (Bottou, 1998) steps, x eventually reaches the global minimum if Q is convex or, if Q is not convex 1 , reach a region where Q is either ﬂat or in some…”
On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimization (2017, background): “see Robbins and Siegmund [1971], Bottou [1998], Nemirovski et al. [2009], Shamir and Zhang [2013], Ghadimi and Lan [2013]). The asymptotic optimal convergence rate of SGD was proved by Chung [1954] and Sacks [1958] to be O(1/N) with twice continuously differentiable and strongly convex objectives. N”
DeltaGrad: Rapid retraining of machine learning models (2020, background): “More recently a lot of work (see e.g., Bottou, 1998; 2003; Zhang, 2004; Bousquet & Bottou, 2008; Bottou, 2010; Bottou et al., 2018) focuses on empirical risk minimization.”
Task-Free Continual Learning (2018, methodology): “We refer to [33, 5] for surveys and overviews on the topic.”
Lifelong Generative Modeling (2017, background): “assumes that the true underlying distribution comes from a single distribution [42].”
Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “We refer to [82], [83] for surveys on the topic.”
No bad local minima: Data independent training error guarantees for multilayer neural networks (2016, background): “However, SGD is only guaranteed to converge to critical points in which the gradient of the expected loss is zero [5], and, speciﬁcally, to stable local minima [25] (this is true also for regular gradient descent [22]).”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, background): “Stochastic gradient descent is an active ﬁeld (Zhang, 2004; Bottou, 1998).”
Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging (2014, background): “[8] Léon Bottou, “Online learning and stochastic approximations,” On-line learning in neural networks, vol.”
Automatic differentiation in machine learning: a survey (2015, methodology): “When introducing new models, machine learning researchers spend considerable effort on the manual derivation of analytical derivatives and subsequently plug these into standard optimization procedures such as L-BFGS (Zhu et al, 1997) or stochastic gradient descent (Bottou, 1998).”
Three Factors Influencing Minima in SGD (2017, neutral): “The standard way of training DNNs involves minimizing a loss function using stochastic gradient descent (SGD) or one of its variants Bottou (1998).”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “Although SGD generally makes for a poor online learning algorithm in the streaming setting for large parametric models such as deep neural networks, we observe that, by (1) meta-training the model for fast adaptation with MAML and (2) employing our online algorithm for probabilistic updates at test ”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “Several works have elaborated on challenges in: online learning in NNs (Heskes and Kappen, 1993), more generally online learning and stochastic approximations (Bottou, 1999; Saad, 1999), or specifically online gradient descent (Zinkevich, 2003), the workhorse of modern optimization.”
Making AI Forget You: Data Deletion in Machine Learning (2019, methodology): “Data Deletion as an Online Problem One convenient way of concretely formulating the computational challenge of data deletion is via the lens of online algorithms [17].”
Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks (2015, methodology): “We perform optimization using stochastic gradient descent (Bottou, 1998).”