An Improved Analysis of Stochastic Gradient Descent with Momentum
By Yanli Liu, Wotao Yin. Neural Information Processing Systems, 2020
320 citations recorded by the source corpus.
SGD with momentum (SGDM) has been widely applied in many machine learning tasks, and it is often applied with dynamic stepsizes and momentum weights tuned in a stagewise manner. Despite of its empirical advantage over SGD, the role of momentum is still unclear in general since previous analyses on SGDM either provide worse convergence bounds than those of SGD, or assume Lipschitz or quadratic objectives, which fail to hold in practice. Furthermore, the role of dynamic parameters have not been addressed. In this work, we show that SGDM converges as fast as SGD for smooth objectives under both strongly convex and nonconvex settings. We also prove that multistage strategy is beneficial for SGDM compared to using fixed parameters. Finally, we verify these theoretical claims by numerical experiments.
Subfields: Artificial Intelligence, Computational Mechanics, Management Science and Operations Research
View Paper: https://www.semanticscholar.org/paper/d9c1cd889f68e464f3745a78ea7a085d8304b14b
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Alexandre D'efossez
- Yoshua Bengio
- L. Eon Bottou
- Jason Aaron Edward Weston
- Francis R. Bach
- Nicolas Usunier
- Martin Jaggi
- Mohit Bansal
In Computational Mechanics
- Alexandre D'efossez
- L. Eon Bottou
- Xinmeng Huang
- Francis R. Bach
- Ping Li
- Nicolas Usunier
- Xiaoyun Li
- Peter Richtárik
In Management Science and Operations Research
- Francis R. Bach
- Xinmeng Huang
- Ping Li
- Nicolas Usunier
- Alexandre D'efossez
- Xiaoyun Li
- L. Eon Bottou
- Diganta Misra
Citations in the corpus
- Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, methodology): “…back to Nesterov’s acceleration (Yurri, 2004) and the heavy-ball method (Polyak, 1964) in deterministic settings, which has been extended to the stochastic scenario (Yan et al., 2018; Yu et al., 2019a; Liu et al., 2020) and other domains (Yuan et al., 2021; He et al., 2023b,a; Chen et al., 2023).”
- A Simple Convergence Proof of Adam and Adagrad (2020, background): “Liu et al. (2020) achieves a similar dependency in (1− β1) as here, but with weaker assumptions.”