Large Scale Online Learning

By L. Eon Bottou, Yann LeCun. 2003

457 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

No more pesky learning rates (2012, methodology): “…SGD has increased in recent years due to the ever-increasing amounts of streaming data, to theoretical optimality results for generalization error (Bottou & LeCun, 2004), and to competitions being won by SGD methods, such as the PAS-CAL Large Scale Learning Challenge (Bordes et al., 2009), where…”
Black Box Variational Inference (2013, methodology): “In stochastic optimization, we maximize a function using noisy estimates of its gradient (Robbins and Monro, 1951; Kushner and Yin, 1997; Bottou and LeCun, 2004).”
Minimizing finite sums with the stochastic average gradient (2013, background): “The most wildly successful class of algorithms for taking advantage of the sum structure for problems where n is very large are stochastic gradient (SG)methods [7,49].”
Variational Inference: A Review for Statisticians (2016, background): “Stochastic optimization has enabled modern machine learning to scale to massive data (Le Cun and Bottou 2004).”
Deep Learning: Methods and Applications (2014, methodology): “Stochastic gradient descend (SGD) algorithms are the most efficient algorithm when the training set is large and redundant as is the case for most applications [39].”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “+ on-line learning + forward transfer – no backward transfer – no knowledge retention – single task/domain (Bottou, 1999) (Bottou and LeCun, 2004) (Cesa-Bianchi and Lugosi, 2006) (Shalev-Shwartz, 2012) (C.”
Lifelong Generative Modeling (2017, background): “The idea of learning in a continual manner has been explored extensively in machine learning, seeded by the seminal works of lifelong-learning [126, 125, 117], online-learning [32, 9, 11, 12] and sequential linear gaussian models [105, 36] such as the Kalman Filter [56] and its non-linear counterpar”
Stochastic dual coordinate ascent methods for regularized loss (2012, neutral): “SGD ﬁnds an (cid:15) P -sub-optimal solution in time ˜ O (1 / ( λ(cid:15) P )) .”
Slow Learners are Fast (2009, background): “Moreover, learning-theory guarantees show that O (1) passes over a dataset suﬃce to obtain optimal estimates (Bottou and LeCun, 2004; Bottou and Bousquet, 2007).”
Optimization with Sparsity-Inducing Penalties (2011, background): “1) may also be tackled with stochastic optimization approaches, which has been the focus of much recent research [21, 23, 108, 129].”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, methodology): “To make deep learning techniques scalable to very large training data, theoretically sound parallel learning algorithms or more effective architectures than the existing ones need to be further developed (e.g., [49, 50, 112, 120, 220]).”
Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization (2009, methodology): “The SGD method has been very popular in the machine learning community due to its capability of scaling with large data sets and good generalization performance observed in practice (e.g., [3] [4]).”
Efficient Distributed Linear Classification Algorithms via the Alternating Direction Method of Multipliers (2012, background): “This few number of the iterations can be enough for large-scale classification tasks since the SVM (or logistic regression) uses loss functions that approximate the misclassification errors, and it may not be necessary to minimize these objectives exactly [1, 17].”
Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach (2018, background): “In particular, these theorems do not provide general guidance on how the error of the network scales with its size at the end of training.”
Optimization for machine learning (2010, background): “1) may also be tackled with stochastic optimization approaches, which has been the focus of much research (Bottou, 1998; Bottou and LeCun, 2004; Shapiro et al., 2009).”
Towards a unified architecture for in-RDBMS analytics (2012, methodology): “[15] L. Bottou and Y. LeCun.”
Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks (2018, background): “Parameter optimization in machine learning typically relies on the stochastic gradient descent algorithm (SGD), which makes an empirical estimate of the gradient of the objective function over a small number of sample points [5].”
A Survey of Optimization Methods From a Machine Learning Perspective (2019, background): “uming. Modern machine learning models often need to analyze and process large-scale data, which is difﬁcult and costly. Stochastic optimization enables machine learning to be extended on massive data [193]. This reminds us of an attractive technique to handle large data sets: stochastic optimization”