Stochastic Gradient Descent Tricks

By L. Eon Bottou. 2012

2,223 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “, random reshuffling, of the training data [12, 38, 41].”
LVIS: A Dataset for Large Vocabulary Instance Segmentation (2019, methodology): “balanced classiﬁcation datasets (e.g., [32]). In CAS, the data sampler ﬁrst selects a category and then an image containing that category. Consistent with repeat factor sampling and SGD bestpractices [2], we iterate over random permutations of categories and within each category random permutations ”
Structured Convolutional Kernel Networks for Airline Crew Scheduling (2021, background): “For instance, Bottou (2012) introduced graph transformer networks trained end-to-end using weighted acyclic directed graphs to represent a sequence of digits in handwritten character recognition.”
Deep Clustering for Unsupervised Learning of Visual Features (2018, methodology): “This cost function is minimized using mini-batch stochastic gradient descent [7] and backpropagation to compute the gradient [34].”
The Hidden Vulnerability of Distributed Learning in Byzantium (2018, background): “Such “errors” are often beneficial, as they may allow the descent process to leave sub–optimal local minima (Bottou, 2012).”
Adaptive Personalized Federated Learning (2020, methodology): “For the learning rate, we use the linear decay structure with respect to the stochastic steps, suggested by Bottou (2012). At each iteration the learning rate is decreased by 1%, unless otherwise stated.”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, neutral): “For a more practically-focused discussion, see [11].”
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, methodology): “Generally, the training of a CNN is achieved by using SGD on a given training dataset through hundreds of epochs (Steps 5-11).”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “Data scientist often use only a subset of the training data or a subset of the available features (Bottou, 2012).”
Survey on Automated Machine Learning (2019, methodology): “Data scientist often use only a subset of the training data or a subset of the available features (Bottou, 2012).”
Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “·i 1), it resembles one large step in the direction of the (noisy) average around the current weights x and Entropy-SGD thus looks similar to averaged SGD in the literature (Polyak & Juditsky, 1992; Bottou, 2012).”
Class-Incremental Instance Segmentation via Multi-Teacher Networks (2021, methodology): “We use SGD (Bottou 2012) to train all the models in the experiments.”
Distributed optimization with arbitrary local solvers (2015, methodology): “We refer the reader to [8, 39–41] for recent theoretical and practical assessment of SGD. Generally speaking, the method is extremely easy to implement, and converges to modest accuracies very quickly, which is often satisfactory in applications in machine learning.”
PANDA: Pose Aligned Networks for Deep Attribute Modeling (2013, methodology): “The whole network is trained jointly by standard backpropagation of the error [24] and stochastic gradient descent [2] using as a loss function the sum of the log-losses of each attribute for each training sample.”
Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms (2015, neutral): “For most problems, this update step is easy to compute, and perhaps because of this SGD is a ubiquitous algorithm with a wide range of applications in machine learning [1], including neural network backpropagation [2, 3, 13], recommendation systems [8, 19], and optimization [20].”