Learning Rate Adaptation in Stochastic Gradient Descent

By Vassilis P. Plagianakos, George D. Magoulas, Michael N. Vrahatis. 2001

55 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Simon Lacoste-Julien
2. Gauthier Gidel
3. Issam Hadj Laradji
4. Peter Richtárik
5. Sharan Vaswani
6. Nicolas Loizou
7. Mark W. Schmidt
8. Atilim Gunes Baydin

In Optimization & Training

1. Simon Lacoste-Julien
2. Issam Hadj Laradji
3. Gauthier Gidel
4. Sharan Vaswani
5. Nicolas Loizou
6. Peter Richtárik
7. Mark W. Schmidt
8. Aaron Mishkin

Citations in the corpus

• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “Techniques for setting the step-size automatically in classic SGD methods include using a meta-learning procedure to modify the main stochastic algorithm [6, 80, 69, 2, 57, 80, 71], heuristics to adjust the learning rate on the fly [38, 17, 64, 68], and recent adaptive methods inspired by online lea”
• Online Learning Rate Adaptation with Hypergradient Descent (2017, methodology): “Plagianakos et al. [19, 18] proposed methods using gradient-related information of up to two previous steps in adapting the learning rate.”
• Convolutional Neural Network Based Automatic Object Detection on Aerial Images (2016, methodology): “Since the network converged and overfitted in the first stage, our approach was to reduce the learning rate and use a more adaptive algorithm [22] to facilitate small changes in the fitted function.”