Note on Learning Rate Schedules for Stochastic Optimization

By C. Darken, John E. Moody. 1990

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Style transfer-based image synthesis as an efficient regularization technique in deep learning (2019, methodology): “popular approach is to use dedicated optimizers such as Momentum, AdaGrad, Adam [7], learning rate schedules [8],”
Differentially Private Model Publishing for Deep Learning (2019, background): “when k < 1, it is known as “search-then-converge” [13], which decreases the noise scale linearly during the SGD search phase when t is less than the “search time” 1/k, and decreases the noise by 1/t when t is greater than 1/k.”
Natural Language Grammatical Inference with Recurrent Neural Networks (2000, methodology): “Weights were initialized as shown in Haykin [26], where weights are initialized on a node by node basis as uniformly distributed random numbers in the range ÿ2:4=Fi; 2:4=Fi , where Fi is the fan-in of neuron i.”
Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming (2006, methodology): “ Darken and Moody (1991) suggests a “search then converge” (STC) procedure for computing the stepsizes, as given by the formula,”
Recursive Least Squares Dictionary Learning Algorithm (2010, methodology): “For the (continuous ) k-means algorithm a Search-Then-Converge scheme has been used with good results [30]–[32].”
Deep neural network with weight sparsity control and pre-training extracts hierarchical features and enhances classification performance: Evidence from whole-brain resting-state functional connectivity patterns of schizophrenia (2015, unknown): “The learning rate α(t) was initially set to 0.002 (i.e., α(0) = 0.002) and then gradually reduced after the first 250 epochs (Bengio, 2013; Darken and Moody, 1992).”
A Survey of Optimization Methods From a Machine Learning Perspective (2019, background): “It is a tricky problem for setting an appropriate value of the learning rate [30], [33], [106].”
Learning to Act Using Real-Time Dynamic Programming (1995, methodology): “Appendix B describes a method for meeting the required learning rate conditions that was developed by Darken and Moody [ 191.”
Neural networks for the prediction and forecasting of water resource variables: a review of modelling issues and applications (2000, methodology): “Other approaches include the so-called search-then-converge schedules (e.g. Darken and Moody, 1990), algorithms that make use of second-order information (e.g. Fahlman, 1988) and a novel approach that dynamically optimises learning rate and momentum (Yu and Chen, 1997).”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “The approach of increasing the mini-batch size in this way is analyzed by Friedlander and Schmidt (2012). Note that for other neural network optimization problems, such as ones involving larger training datasets than these autoencoder problems, a more slowly increasing schedule, or one that stops in”
Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference (2001, background): “The RNN learning rate was linearly reduced over the training period (see [11] regarding learning rate schedules) from an initial value of 0.”