Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks

By Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, Jonathan M. Cohen. 2019

108 citations recorded by the source corpus.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

1. Nitish Shirish Keskar
2. Caiming Xiong
3. Richard Socher
4. Boris Ginsburg
5. Bryan McCann
6. Oleksii Kuchaiev
7. Vitaly Lavrukhin
8. Stanislav Beliaev

Citations in the corpus

• CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “We also experimented with explicit memory-saving optimizers including SM3 (Anil et al., 2019), Adafactor (Shazeer & Stern, 2018), and NovoGrad (Ginsburg et al., 2019) with mixed results.”
• Quartznet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions (2019, methodology): “The models are trained using NovoGrad optimizer [26] with a co-sine annealing learning rate policy.”