Train faster, generalize better: Stability of stochastic gradient descent

By Moritz Hardt, B. Recht, Yoram Singer. 2015

1,448 citations recorded by the source corpus.

We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimization. We derive stability bounds for both convex and non-convex optimization under standard Lipschitz and smoothness assumptions. Applying our results to the convex case, we provide new insights for why multiple epochs of stochastic gradient methods generalize well in practice. In the non-convex case, we give a new interpretation of common practices in neural networks, and formally show that popular techniques for training large deep models are indeed stability-promoting. Our findings conceptually underscore the importance of reducing training time beyond its obvious benefit.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Percy Liang
  3. Sergey Levine
  4. Mohit Bansal
  5. Quoc V. Le
  6. Ruslan Salakhutdinov
  7. Dacheng Tao
  8. Jianfeng Gao

In Computational Mechanics

  1. Peter Richtárik
  2. Nicolas Loizou
  3. Jason D. Lee
  4. Tengyu Ma
  5. Simon Lacoste-Julien
  6. Yuanzhi Li
  7. Michael I. Jordan
  8. Nati Srebro

In Optimization & Training

  1. Peter Richtárik
  2. Tengyu Ma
  3. Yoshua Bengio
  4. Jason D. Lee
  5. Nicolas Loizou
  6. Yuanzhi Li
  7. Simon Lacoste-Julien
  8. Martin Jaggi

In Scaling & Learning Theory

  1. Tengyu Ma
  2. Yuanzhi Li
  3. Jason D. Lee
  4. Behnam Neyshabur
  5. Colin Wei
  6. Samet Oymak
  7. Quanquan Gu
  8. Simon Shaolei Du

Citations in the corpus