MaSS: an Accelerated Stochastic Method for Over-parametrized Learning

By Chaoyue Liu, Mikhail Belkin. 2018

19 citations recorded by the source corpus.

In this paper we introduce MaSS (Momentum-added Stochastic Solver), an accelerated SGD method for optimizing over-parameterized networks. Our method is simple and efficient to implement and does not require changing parameters or computing full gradients in the course of optimization. We provide a detailed theoretical analysis for convergence and parameter selection including their dependence on the mini-batch size in the quadratic case. We also provide theoretical convergence results for a more general convex setting. We provide an experimental evaluation showing strong performance of our method in comparison to Adam and SGD for several standard architectures of deep networks including ResNet, convolutional and fully connected networks. We also show its performance for convex kernel machines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Simon Lacoste-Julien
2. Peter Richtárik
3. Gauthier Gidel
4. Nicolas Loizou
5. Issam Hadj Laradji
6. Francis R. Bach
7. Sharan Vaswani
8. Mark W. Schmidt

In Computer Vision and Pattern Recognition

1. Simon Lacoste-Julien
2. Mark W. Schmidt
3. Gauthier Gidel
4. Issam Hadj Laradji
5. Francis R. Bach
6. Luc Van Gool
7. Andrew Zisserman
8. Sharan Vaswani

In Optimization & Training

1. Simon Lacoste-Julien
2. Peter Richtárik
3. Gauthier Gidel
4. Nicolas Loizou
5. Issam Hadj Laradji
6. Sharan Vaswani
7. Mark W. Schmidt
8. Francis R. Bach

Citations in the corpus

• Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, background): “Indeed, recent work [77, 67, 46, 5, 42, 11, 28] shows that classic SGD (with a constant step-size) and its accelerated variants can achieve the full-gradient convergence rates without VR under the assumption that the model is expressive enough to interpolate the data.”
• Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron (2018, methodology): “[20] also considered minimizing strongly-convex loss functions using a variant of Nesterov acceleration assuming interpolation.”
• On the Generalization Benefit of Noise in Stochastic Gradient Descent (2020, unknown): “Some authors argue that SGD with Momentum significantly outperforms vanilla SGD (Sutskever et al., 2013), while others claim that SGD with and without Momentum are equivalent if one maintains a constant “effective learning rate” (Mandt et al., 2017; Kidambi et al., 2018; Liu & Belkin, 2018a).”