No more pesky learning rates

By Tom Schaul, Sixin Zhang, Yann LeCun. 2012

495 citations recorded by the source corpus.

The performance of stochastic gradient descent (SGD) depends critically on how learning rates are tuned and decreased over time. We propose a method to automatically adjust multiple learning rates so as to minimize the expected error at any one time. The method relies on local gradient variations across samples. In our approach, learning rates can increase as well as decrease, making it suitable for non-stationary problems. Using a number of convex and non-convex learning tasks, we show that the resulting algorithm matches the performance of SGD or other adaptive approaches with their best settings obtained through systematic search, and effectively removes the need for learning rate tuning.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Adam: A Method for Stochastic Optimization (2014, methodology): “Other stochastic optimization methods include vSGD (Schaul et al., 2012), AdaDelta (Zeiler, 2012) and the natural Newton method from Roux & Fitzgibbon (2010), all setting stepsizes by estimating curvature from ﬁrst-order information.”
ADADELTA: An Adaptive Learning Rate Method (2012, methodology): “A recent method by Schaul et al. [6] incorporating the diagonal Hessian with ADAGRAD-like terms has been introduced to alleviate the need for hand speciﬁed learning rates.”
An Empirical Model of Large-Batch Training (2018, background): “A simple model appearing in [SZL13] suggests treating the per-example loss L i as a shifted version of the true loss, L i ( θ ) = L ( θ − c i ) , where c i is a random variable with mean zero and covariance matrix Σ c .”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, methodology): “3.1 Problem Setup We now introduce the noisy quadratic model [Schaul et al., 2013, Martens, 2014, Wu et al., 2018], where the true function being optimized is a convex quadratic.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “These techniques include using meta-learning procedures to modify the main stochastic algorithm [2, 6, 63, 75, 77, 86, 86], heuristics to adjust the learning rate on the fly [20, 43, 70, 74], and recent adaptive methods inspired by online learning [21, 39, 51, 60, 67, 68, 87].”
Reducing Noise in GAN Training with Variance Reduced Extragradient (2019, methodology): “This adaptive variant of Adam is motivated by the step size η∗ = η 2 t vt derived by Schaul et al. (2013). (VRAd) is simply the square-root of η∗ in order to stick with Adam’s scaling of vt.”
Prioritized Experience Replay (2015, background): “…replaying a transition – this can be effective if the underlying optimizer employs adaptive step-sizes that reduce gradients in high-noise directions (Schaul et al., 2013; Kingma & Ba, 2014), thus placing the burden of distinguishing between learnable and unlearnable transitions on the optimizer.”
RNADE: The real-valued neural autoregressive density-estimator (2013, methodology): “The gradient descent learning rate can be adjusted automatically using, for example, the techniques developed by Schaul et al. [30].”
Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “This kind of exponentially decaying averaging scheme is commonly used in methods involving diagonal or block-diagonal approximations (with much smaller blocks than ours) to the curvature matrix (e.g. LeCun et al., 1998; Park et al., 2000; Schaul et al., 2013).”
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, background): “More involved optimal step size choices involve some kind of second order information that can be obtained from the Hessian of the loss function [Schaul et al., 2013].”
Neural Optimizer Search with Reinforcement Learning (2017, neutral): “…this difﬁculty (e.g., Riedmiller & Braun (1992); LeCun et al. (1998); Schraudolph (2002); Martens (2010); Le et al. (2011); Duchi et al. (2011); Zeiler (2012); Martens & Sutskever (2012); Schaul et al. (2013); Pascanu & Bengio (2013); Pascanu et al. (2013); Kingma & Ba (2015); Ba et al. (2017)).”
Recent advances in convolutional neural networks (2015, background): “Additionally, learning rate schedules [110, 111] have been proposed to adjust the learning rate during the training.”
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, methodology): “Like Quasi-Newton methods, several methods adaptively adjust learning rates based on local curvature [2, 40], while ADAGRAD [11] applies different learning rates to different dimensions.”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “There are also slightly more sophisticated methods (Schaul et al., 2013; Zeiler, 2013) which use preconditioners that combine the diagonal of F̄ with other quantities (such as an approximation of the diagonal of the Gauss-Newton/Fisher in the case of Schaul et al.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “ing rate procedures have been proposed recently and merit more attention and evaluations in the neural network context, such as adagrad (Duchi et al., 2011) and the adaptive learning rate method from Schaul et al. (2012) which claims to remove completely the need for a learning rate hyper-parameter.”
Lookahead Optimizer: k steps forward, 1 step back (2019, neutral): “While simple, this model is a proxy for neural network optimization and effectively optimizing it remains a challenging open problem [37, 26, 43, 47].”
Cyclical Learning Rates for Training Neural Networks (2015, background): “[22] discuss an adaptive learning rate based on a diagonal estimation of the Hessian of the gradients.”
Automatic differentiation in machine learning: a survey (2015, methodology): “The convergence rate of gradient-based methods is usually improved by adaptive step-size techniques that adjust the step size η on every iteration (Duchi et al., 2011; Schaul et al., 2013; Kingma and Ba, 2015).”