Accelerating SGD with momentum for over-parameterized learning

95 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

On the Generalization Benefit of Noise in Stochastic Gradient Descent (2020, unknown): “Some authors argue that SGD with Momentum significantly outperforms vanilla SGD (Sutskever et al., 2013), while others claim that SGD with and without Momentum are equivalent if one maintains a constant “effective learning rate” (Mandt et al., 2017; Kidambi et al., 2018; Liu & Belkin, 2018a).”
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks (2020, methodology): “To give just one example, we expect that accelerated methods, such as the Nesterov’s method [26] and its stochastic gradient extensions in the over-parameterized case [22, 35] to have faster convergence rates for non-linear systems in terms of the condition numbers deﬁned in this work.”
On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings (2020, methodology): “…guarantees for existing momentum schemes, such as Polyak’s and Nesterov’s, have led many authors to develop alternative accelerated methods speciﬁcally for use with stochastic gradients Lan (2012); Allen-Zhu (2017); Kidambi et al. (2018); Kulunchakov & Mairal (2019); Liu & Belkin (2020).”