Large-Scale Machine Learning with Stochastic Gradient Descent

By L. Eon Bottou. 2010

6,396 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, background): “While improving the speed of current methods is crucial to work on the very large datasets currently available (Bottou, 2010), it is also vital to develop novel approaches able to tackle new problems.”
On Empirical Comparisons of Optimizers for Deep Learning (2019, background): “In most deep learning applications, the cost of evaluating ∇ (cid:96) scales linearly with the data set size and it is usually more effective to use a stochastic estimator of ∇ (cid:96) , whose cost is constant in the data set size (Bottou, 2010).”
Revisiting Semi-Supervised Learning with Graph Embeddings (2016, methodology): “We adopt stochastic gradient descent (SGD) (Bottou, 2010) to train our model in the mini-batch mode.”
On Sampling Strategies for Neural Network-based Collaborative Filtering (2017, methodology): “links # tf # tд # ti pointwise pairwise IID [3] b bk b (1 + k ) b (1 + k ) b (1 + k ) vec ✓ × Negative [1, 21, 29] b bk b b (1 + k ) b (1 + k ) vec ✓ ✓ Stratified (by Items) b bk b (1 + k ) s b (1 + k ) vec ✓ × Negative Sharing b b (b − 1) b b b × b mat ✓ ✓”
Differentiable Causal Discovery from Interventional Data (2020, background): “On the other hand, DCDI is not greatly affected by the sample size since it relies on stochastic gradient descent which is known to scale well with the data set size [3].”
Reducing Noise in GAN Training with Variance Reduced Extragradient (2019, methodology): “These iterative methods handle large-scale training datasets by computing gradient estimates on a subset of it, a mini-batch, instead of using all the samples at each step, the full batch, resulting in a method called stochastic gradient descent (SGD, Robbins and Monro (1951); Bottou (2010)). SGD me”
Variance Reduced Stochastic Gradient Descent with Neighbors (2015, background): “Stochastic gradient descent (SGD) is a popular alternative, in particular in the context of large-scale learning [2, 10].”
Towards the Science of Security and Privacy in Machine Learning (2016, methodology): “They formulate a min-max problem between the adversary applying perturbations to each training point to maximize the model’s classiﬁcation error, and the learning procedure attempting to minimize the model’s misclassiﬁcation error: They solve the problem using stochastic gradient descent [89].”
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016, background): “In (Li et al., 2014), the authors argue that the convergence rate of SGD for the large-batch setting can be improved by obtaining iterates through the following proximal sub-problem. x k +1 = arg min (5) The motivation for this strategy is, in the context of large-batch methods, to better utilize a ”
Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017, methodology): “s were conducted with batch of 2048. 8 described above. The main difference between our works is the use of a linear scaling of the learning rate9, similarly to Krizhevsky (2014), and as suggested by Bottou (2010). However, we found that linear scaling works less well on CIFAR10, and later work foun”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015, methodology): “The ability to train end-to-end in order to jointly optimise all the weights in the network using an efficient weight update technique such as stochastic gradient descent (SGD) [17] is an additional benefit since it is more easily repeatable.”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “spect, the eect of averaging has been characterized as being similar to that of using a second-order method in which the Hessian approximations approach the Hessian of the objective at the minimizer [122,20]; see x6.2.1.) This asymptotic behavior is dicult to achieve in practice, but is possible in ”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, neutral): “When the gradients are computationally expensive, one can alternatively use it’s stochastic version (SGD) that replaces the above gradient with the gradient of averages of losses over subsets (such a subset will be called the mini-batch ) of D (see [Bottou, 2010] for a classical reference).”
Sharp Minima Can Generalize For Deep Nets (2017, methodology): “These correspond, respectively, to low approximation , optimization and estimation error as described by Bottou (2010).”
Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs (2020, methodology): “In order to generate a learning model using (1), the commonly used mechanism is Stochastic Gradient Descent (SGD) [Bot10].”
rTop-k: A Statistical Estimation Approach to Distributed SGD (2020, neutral): “SGD methods [1], [3]–[15], as well as other fundamental contributions to distributed first-order [16]–[20] and secondorder [21]–[25] optimization procedures.”
Decentralized Deep Learning with Arbitrary Communication Compression (2019, background): “Algorithms based on stochastic gradient descent (SGD) (Robbins and Monro, 1951; Bottou, 2010) and its mini-batch variant (Dekel et al.”
Data Encoding for Byzantine-Resilient Distributed Optimization (2019, methodology): “Due to its simplicity and remarkable empirical performance, SGD has become arguably the most widely-used optimization algorithm in many large-scale applications, especially in deep learning [Bot10,RSS12,DCM + 12].”
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations (2019, background): “STOCHASTIC Gradient Descent (SGD) [2] and its many variants have become the workhorse for modern largescale optimization as applied to machine learning [3], [4].”