Why Momentum Really Works

By Gabriel Goh. 2017

191 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Optimization & Training

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, background): “In poorly conditioned optimization problems, parameter values often oscillate along the large-curvature directions rather than decreasing in a predictable way (see e.g. [Goh17] and Appendix E.1).”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, methodology): “Following the existing treatment of the deterministic version of the algorithm [Chiang, 1974, Qian, 1999, Yang et al., 2018, Goh, 2017], we divide our analysis two cases: overdamping and underdamping.”
On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization (2018, background): “3 Optimal learning rate for gradient descent on quadratic objec-tive does not depend on current parameter value ( cf. Goh (2017)). network, induces on gradient descent a certain preconditioning scheme.”
signSGD: compressed optimisation for non-convex problems (2018, background): “Existing theoretical analyses of momentum often rely on the absence of gradient stochasticity (e.g. Jin et al. (2017)) or convexity (e.g. Goh (2017)) to show that momentum’s asymptotic convergence rate can beat gradient descent.”
Lookahead Optimizer: k steps forward, 1 step back (2019, methodology): “Here we analyze the quadratic model without noise using gradient descent with momentum [32, 8] and show that when the system is under-damped, Lookahead is able to improve on the convergence rate.”
Dive into Deep Learning (2020, background): “, the beautiful expository article136 by (Goh, 2017) for an indepth analysis and interactive animation.”
Semi-Supervised Learning by Label Gradient Alignment (2019, methodology): “From Equation 5 we obtain an alternate formula for θ ∗ : The iterates θ k admit the following closed form (Goh, 2017): We would like to understand the effect of label gradient alignment when applied in the linear regression setting.”
Understanding the Role of Momentum in Stochastic Gradient Methods (2019, background): “3 For example [8] says “So set β as close to 1 as you can, and then ﬁnd the highest α which still converges.”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, methodology): “These methods are typically optimized with SGD variants [329]–[331], which, together with GPU parallelization, makes them scale to large data sets.”
Adaptive Gradient Methods at the Edge of Stability (2022, background): “Stability threshold Reference for threshold common knowledge, reproduced in [8] for completeness 2+2 β 1 η [16], reproduced in [8] for completeness this paper S TANDARD N ESTEROV ( η, β 1 ) [8] E MA N ESTEROV ( η, β 1 ) this paper Table 1: Gradient descent algorithms and their stability thresholds.”
Aggregated Momentum: Stability Through Passive Damping (2018, methodology): “However, large β values are prone to oscillations and instability [21, 4], requiring a smaller learning rate and hence slower convergence.”
Explaining neural scaling laws (2021, background): “In the supplement, we provide an in-depth derivation of this behavior and expressions for the leading contributions to L ( D, P ) − L ( P ) and L ( D, P ) − L ( D ).”
On the Generalization Benefit of Noise in Stochastic Gradient Descent (2020, unknown): “This allows Momentum to minimize the training loss in fewer steps than full batch gradient descent (Goh, 2017).”
Rethinking the Hyperparameters for Fine-tuning (2020, unknown): “Momentum is widely used for accelerating and smoothing the convergence of SGD by accumulating a velocity vector in the direction of persistent loss reduction (Polyak, 1964; Sutskever et al., 2013; Goh, 2017).”
Pipelined Backpropagation at Scale: Training Large Models without Batches (2020, methodology): “…for large momentum values • Restore the beneﬁts of momentum for poorly conditioned losses We follow a similar approach as (Odonoghue & Candes, 2015; Goh, 2017) and write the loss in terms of an eigenbasis of the quadratic as: where φ = [ φ (1) , ..., φ ( N ) ] T correspond to the parameters being…”
Adaptive norms for deep learning with regularized Newton methods (2019, background): “2 It is well known that GD struggles to progress towards the minimizer of quadratics along low-curvature directions (see e.g., Goh (2017)).”
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability (2021, background): “In the leftmost plot, the vertical dotted line marks the iteration where the sharpness first crosses the MSS. (Note that unlike vanilla gradient descent, momentum gradient descent can sometimes cause the train loss to increase even when the algorithm is stable (Goh, 2017).)”
MakeSense: Automated Sensor Design for Proprioceptive Soft Robots (2019, unknown): “To determine the undeformed and deformed coordinates of sample points in 3D, we interpolate the nodal degrees of freedom with Lagrange shape functions (see our appendix for details).”