Understanding the Role of Momentum in Stochastic Gradient Methods

By Igor Gitman, Hunter Lang, Pengchuan Zhang, Lin Xiao. 2019

118 citations recorded by the source corpus.

The use of momentum in stochastic gradient methods has become a widespread practice in machine learning. Different variants of momentum, including heavy-ball momentum, Nesterov's accelerated gradient (NAG), and quasi-hyperbolic momentum (QHM), have demonstrated success on various tasks. Despite these empirical successes, there is a lack of clear understanding of how the momentum parameters affect convergence and various performance measures of different algorithms. In this paper, we use the general formulation of QHM to give a unified analysis of several popular algorithms, covering their asymptotic convergence conditions, stability regions, and properties of their stationary distributions. In addition, by combining the results on convergence rates and stationary distributions, we obtain sometimes counter-intuitive practical guidelines for setting the learning rate and momentum parameters.

Subfields: Artificial Intelligence, Mathematical Physics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianyu Wang
2. Michael G. Rabbat
3. Nicolas Ballas
4. Mahmoud Assran
5. Yanli Liu
6. Wotao Yin
7. Vinayak Tantia
8. Gauri Joshi

In Mathematical Physics

1. Michael G. Rabbat
2. Mahmoud Assran
3. Wotao Yin
4. Stanley J. Osher
5. Ming Yan

In Statistics and Probability

1. Michael G. Rabbat
2. Yanli Liu
3. Nicolas Ballas
4. Jianyu Wang
5. Mahmoud Assran
6. Wotao Yin
7. Ali H. Sayed
8. Angelia Nedić

Citations in the corpus

• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, methodology): “In the setting of smooth non-convex functions, Gitman et al. (2019) establish stability and asymptotic convergence results for the quasi-hyperbolic momentum method (Ma & Yarats, 2019), which can be viewed as interpolating between SGD and a stochastic momentum method.”
• On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings (2020, methodology): “Gitman et al. (2019) characterize the stationary distribution of the Quasi-Hyperbolic Momentum (QHM) method (Ma & Yarats, 2019) around the minimizer for strongly-convex quadratic functions with bounded gradients and bounded gradient noise variance.”
• An Improved Analysis of Stochastic Gradient Descent with Momentum (2020, background): “, without convergence rate) [9], which requires the momentum weights to approach either 0 or 1, and therefore contradicts the common practice in neural network training.”