L4: Practical loss-based stepsize adaptation for deep learning

By Michal Rolinek, Georg Martius. 2018

68 citations recorded by the source corpus.

We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by strongly improving the performance of Adam and Momentum optimizers. The enhanced optimizers with default hyperparameters consistently outperform their constant stepsize counterparts, even the best ones, without a measurable increase in computational cost. The performance is validated on multiple architectures including ResNets and the Differential Neural Computer. A prototype implementation as a TensorFlow optimizer is released.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence (2020, methodology): “In Rolinek and Martius (2018), the L4 algorithm has been proposed showing that a stochastic variant of the Polyak step for SGD achieves good empirical results for training neural networks.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “We compare our methods against Adam [39], which is the most common adaptive method, and other methods that report better performance than Adam: coin-betting [60], L41 [68], and Adabound [51].”
Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize (2021, background): “…model such as a deep neural network (Ma et al., 2018) or involves solving a consistent linear system (Loizou & Richtárik, 2020b;a) or problems such as deep matrix factorization (Rolinek & Martius, 2018; Vaswani et al., 2019b), each individual loss function f i attains its minimum at x ∗ .”
SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation (2020, methodology): “This includes problems such as deep matrix factorization (Rolinek and Martius, 2018; Rahimi and Recht, 2017), binary classification using kernels (Loizou et al.”
DeepOBS: A Deep Learning Optimizer Benchmark Suite (2019, background): “…Hutter, 2017; Reddi et al., 2018), incorporation of second-order information (Martens, 2010; Martens & Grosse, 2015; Botev et al., 2017; Zhang et al., 2017; Chen et al., 2018), and automating optimization (Schaul et al., 2013b; Mahsereci & Hennig, 2017; Rolinek & Martius, 2018), to name just a few.”