A Unified Analysis of Stochastic Momentum Methods for Deep Learning

By Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, Yi Yang. 2018

140 citations recorded by the source corpus.

Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochastic gradient (SG) method, and the stochastic momentum methods including two famous variants, i.e., the stochastic heavy-ball (SHB) method and the stochastic variant of Nesterov?s accelerated gradient (SNAG) method. We propose a framework that unifies the three variants. We then derive the convergence rates of the norm of gradient for the non-convex optimization problem, and analyze the generalization performance through the uniform stability approach. Particularly, the convergence analysis of the training objective exhibits that SHB and SNAG have no advantage over SG. However, the stability analysis shows that the momentum term can improve the stability of the learned model and hence improve the generalization performance. These theoretical insights verify the common wisdom and are also corroborated by our empirical analysis on deep learning.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinmeng Huang
2. Ping Li
3. Xiaoyun Li
4. Wotao Yin
5. Rong Jin
6. Yanli Liu
7. Hao Yu
8. Tianbao Yang

In Computational Mechanics

1. Tianbao Yang
2. Xinmeng Huang
3. Ping Li
4. Xiaoyun Li
5. Rong Jin
6. Fangyu Zou
7. Yanli Liu
8. Yi Xu

Citations in the corpus

• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “In practice, momentum SGD is more commonly used to train deep neural networks since it often can converge faster and generalize better (Krizhevsky et al., 2012; Sutskever et al., 2013; Yan et al., 2018).”
• Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, methodology): “…back to Nesterov’s acceleration (Yurri, 2004) and the heavy-ball method (Polyak, 1964) in deterministic settings, which has been extended to the stochastic scenario (Yan et al., 2018; Yu et al., 2019a; Liu et al., 2020) and other domains (Yuan et al., 2021; He et al., 2023b,a; Chen et al., 2023).”
• An Improved Analysis of Stochastic Gradient Descent with Momentum (2020, methodology): “Previous analysis such as [25] and [9] make the blanket assumption of bounded∇f to circumvent this difficulty.”
• First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time (2017, methodology): “, mini-batch SGD and stochastic momentum (SM) methods) have been analyzed for stochastic non-convex optimization [6, 7, 8, 22].”
• Stochastic Gradient Descent with Polyak’s Learning Rate (2019, background): “When SGD is combined with momentum [17,19] empirical performance is improved [31].”
• On the Convergence of Weighted AdaGrad with Momentum for Training Deep Neural Networks (2018, methodology): “In [41], a three steps version of unified stochastic momentum method has been provided  The convergence rate of above iteration has been established if where y s 0 = x 0 .”