Rethinking learning rate schedules for stochastic optimization

By Rong Ge, Sham M. Kakade, Rahul Kidambi, Praneeth Netrapalli. 2018

14 citations recorded by the source corpus.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

1. Mirco Ravanelli
2. Yoshua Bengio
3. Pawel Swietojanski
4. Moritz Hardt
5. João Monteiro
6. Santiago Pascual
7. Tijana Zrnic
8. Frank Hutter

Citations in the corpus

• Multi-Task Self-Supervised Learning for Robust Speech Recognition (2020, positive): “All the neural networks are optimized with Adam [29], using an initial learning rate of 10−3 which is annealed using a polynomial scheduler [30].”
• Natural Analysts in Adaptive Data Analysis (2019, background): “5, 1] (Robbins & Monro, 1951), or the more recent schemes which cut the learning rate by a constant factor in every so-called epoch, which implies an essentially exponential decay (Hazan & Kale, 2014; Ge et al., 2018).”