No More Pesky Learning Rate Guessing Games

By Leslie N. Smith. 2015

163 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, background): “(2017) proposed Snapshot ensemble, in which a single model is trained by cyclic learning rates (Loshchilov & Hutter, 2016; Smith, 2015) so that it is encouraged to visit multiple local minima.”
Snapshot Ensembles: Train 1, get M for free (2017, background): “nly when the model reaches a minimum. We believe this key insight allows us to leverage more power from our ensembles. Our work is inspired by the recent ﬁndings of Loshchilov & Hutter (2016) and Smith (2016), who show that cyclic learning rates can be effective for training convolutional neural”
SpotFake: A Multi-modal Framework for Fake News Detection (2019, methodology): “Cycling learning rate proposed in [26] is used to find an optimal learning rate for our models.”
Backtracking gradient descent method for general $C^1$ functions, with applications to Deep Learning (2018, methodology): “0 mini-batches), take the mean value of obtained learning rates with rescaling justication and use it in Standard GD for next training iterations. This method is inspired from cyclic learning rates ([42]) and fastai learning rate nder ([51],[54]) (both require manual interference for the learning ra”
A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay (2018, methodology): “In particular, cyclical learning rates were introduced by Smith (2015) and later updated in Smith (2017).”
SGDR: Stochastic Gradient Descent with Restarts (2016, background): “Smith (2015) recently introduced cyclical learning rates for deep learning, his approach is closelyrelated to our approach in its spirit and formulation but does not focus on restarts.”
Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates (2017, methodology): “In this work, we use cyclical learning rates (CLR) and the learning rate range test (LR range test) which were first suggested by Smith (2015) and later updated in Smith (2017). To use CLR, one specifies minimum and maximum learning rate boundaries and a stepsize. The stepsize is the number of itera”
The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints (2018, methodology): “…a single epoch of training, so we either utilize a ﬁxed reasonable initial learning rate such as 1.0, and for longer ﬁnal training runs we utilized either a triangular or exp range (gamma=0.999998) cyclical learning rate [46] with a cycle period of 8 epochs, maximum of 2.0, and minimum of 10 − 5 .”
NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results (2017, methodology): “The training employed the cyclical learning rate strategy [29] and inspired by [37] used both train images and re-scaled images”
fastai: A Layered API for Deep Learning (2020, background): “Other libraries often provide help for grid search or AutoML to guess the best value, but the fastai library implements the learning rate ﬁnder [27] which much more quickly provides the best value for this parameter after a mock training.”
Super-convergence: very fast training of neural networks using large learning rates (2018, unknown): “Figure reproduced from Smith [2017] with permission.”