On the influence of momentum acceleration on online learning
By Kun Yuan, Bicheng Ying, Ali H. Sayed. 2016
67 citations recorded by the source corpus.
This paper examines the convergence rate and mean-square-error performance of momentum stochastic gradient methods in the constant step-size and slow adaptation regime. The results establish that momentum methods are equivalent to the standard stochastic gradient method with a re-scaled (larger) step-size value. The equivalence result is established for all time instants and not only in steady-state. The analysis is carried out for general risk functions, and is not limited to quadratic risks. One notable conclusion is that the well-known benefits of momentum constructions for deterministic optimization problems do not necessarily carry over to the stochastic setting when gradient noise is present and continuous adaptation is necessary. The analysis suggests a method to enhance performance in the stochastic setting by tuning the momentum parameter over time.
Subfields: Optimization & Training, Statistical Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Mikhail Belkin
- Chaoyue Liu
- Sham M. Kakade
- Praneeth Netrapalli
- Rahul Kidambi
- Prateek Jain
- Aaron Sidford
- Anirbit Mukherjee
In Statistical Learning Theory
- Prateek Jain
- Sham M. Kakade
- Mikhail Belkin
- Praneeth Netrapalli
- Rahul Kidambi
- Aaron Sidford
- Soham De
- Lihong Li
Citations in the corpus
- Convergence Guarantees for RMSProp and ADAM in Non-Convex Optimization and an Empirical Comparison to Nesterov Acceleration (2018, background): “[46] Kun Yuan, Bicheng Ying, and Ali H Sayed.”
- Accelerating SGD with momentum for over-parameterized learning (2018, background): “[22] Kun Yuan, Bicheng Ying, and Ali H Sayed.”
- Accelerating Stochastic Gradient Descent (2017, background): “In essence, this paper presents the first known provable analysis of the claim that fast gradient methods are stable when dealing with statistical errors, in contrast to negative results in statistical and non-statistical settings (Paige, 1971; Proakis, 1974; Polyak, 1987; Greenbaum, 1989; Roy and S”
- On the Generalization Benefit of Noise in Stochastic Gradient Descent (2020, unknown): “Many previous works have established that SGD with and without Momentum are equivalent in the small learning rate limit when m is fixed (Orr & Leen, 1994; Qian, 1999; Yuan et al., 2016).”