Momentum Improves Normalized SGD

By Ashok Cutkosky, Harsh Mehta. 2020

196 citations recorded by the source corpus.

We provide an improved analysis of normalized SGD showing that adding momentum provably removes the need for large batch sizes on non-convex objectives. Then, we consider the case of objectives with bounded second derivative and show that in this case a small tweak to the momentum formula allows normalized SGD with momentum to find an $\epsilon$-critical point in $O(1/\epsilon^{3.5})$ iterations, matching the best-known rates without accruing any logarithmic factors or dependence on dimension. We also provide an adaptive method that automatically improves convergence rates when the variance in the gradients is small. Finally, we show that our method is effective when employed on popular large scale tasks such as ResNet-50 and BERT pretraining, matching the performance of the disparate methods used to get state-of-the-art results on both tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, background): “An interesting future direction would be to adapt the algorithm and analysis of [3], which does fit the framework of MIME.”
Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, methodology): “For analysis, we make several mild assumptions used in many works, e.g., [31], [32], [33], [34], [35], [37], [38], [49], [71] Assumption 1 ( L -smoothness): Assumption 2 (Unbiased and bounded gradient oracle): The stochastic gradient oracle g k = E ζ [ ∇ f ( θ k , ζ )] + ξ k is unbiased, i.e., E (…”