Stochastic Gradient Learning in Neural Networks

By L. Eon Bottou. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “The second approach is to use a variant of Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951; Bottou, 1991).”
DeepWalk: online learning of social representations (2014, methodology): “The learning rate α for SGD is initially set to 2.5% at the beginning of the training and then decreased linearly with the number of vertices that are seen so far.”
Natural Language Processing (Almost) from Scratch (2011, methodology): “Maximizing (8) with stochastic gradient (Bottou, 1991) is achieved by iteratively selecting a random example (x, y) and making a gradient step: θ ←− θ + λ ∂ log p(y |x, θ) ∂θ , (17) where λ is a chosen learning rate.”
Multi-Task Learning for Multiple Language Translation (2015, methodology): “The optimization approach we use is the mini-batch stochastic gradient descent approach (Bottou, 1991).”
Optimization Methods for Large-Scale Machine Learning (2016, background): “ equation _w= r F(w). Stochastic approximation theory has had a major impact in signal processing and in areas closer to the subject of this paper, such as pattern recognition [4] and neural networks [20]. After receiving his PhD, Herbert Robbins became a lecturer at New York University, where he co”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, background): “Bottou [1991] points out that large eigenvalues of the Hessian of the loss can create the illusion of a local minima and GD can get stuck there, it further claims that the help of the inherent noise in SGD may help getting out of this obstacle. The origin of this observation is due to Bourrely [1989”
Towards Better Analysis of Deep Convolutional Neural Networks (2016, methodology): “This is usually achieved with stochastic gradient descent [7], an optimization method that first calculates the gradient of the loss function with respect to the weight of each edge in the network and then updates the weight according to the computed gradient.”
Real-Time Federated Evolutionary Neural Architecture Search (2020, methodology): “Then the local model parameters are updated by the batch stochastic gradient descent (SGD) algorithm [23], where b is the local learning batch size.”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, methodology): “In practice, algorithms such as stochastic gradient descent [111] and variants of it such as Adam [112] are used for this optimization.”
Adding Gradient Noise Improves Learning for Very Deep Networks (2015, methodology): “By adding even more stochasticity, this technique gives the model more chances to escape local minima (see a similar argument in Bottou (1992)), or to traverse quickly through the “transient” plateau phase of early learning (see a similar analysis for momentum in Sutskever et al. (2013)).”
Recurrent Convolutional Neural Networks for Text Classification (2015, methodology): “We use stochastic gradient descent (Bottou 1991) to optimize the training target.”
Stochastic Variance Reduction for Nonconvex Optimization (2016, background): “Sgd dates at least to the seminal work (Robbins & Monro, 1951); and since then it has been developed in several directions (Poljak & Tsypkin, 1973; Ljung, 1977; Bottou, 1991; Kushner & Clark, 2012).”
Word-Based Dialog State Tracking with Recurrent Neural Networks (2014, methodology): “The RNNs are trained using Stochastic Gradient Descent (SGD), maximizing the log probability of the sequences of observed beliefs in the training data (Bottou, 1991).”
Assessment and Validation of Machine Learning Methods for Predicting Molecular Atomization Energies. (2013, background): “59 Typically the training procedure requires a large number of iterations, but scales well with the numbers of samples.”
Efficient Decentralized Deep Learning by Dynamic Model Averaging (2018, positive): “This allows using the common mini-batch approach [3] for training deep neural networks.”
An empirical analysis of the optimization of deep network loss surfaces (2016, background): “They are usually minimized using first-order gradient descent algorithms such as stochastic gradient descent (SGD) [3].”
Conversational Contextual Cues: The Case of Personalization and History for Response Ranking (2016, methodology): “The hidden layer parameters, the ngram embed-dings, and the user embeddings are trained jointly, and we use stochastic gradient descent (SGD) to optimize the parameters of our models (Bottou, 1991).”
High-Accuracy Low-Precision Training (2018, background): “Many machine learning training tasks can be written as an optimization problem over a ﬁnite sum of N components minimize f ( w ) = 1 A standard way of solving these optimization problems over very large training datasets is by using stochastic gradient descent (SGD) [4, 5, 25].”