SGDR: Stochastic Gradient Descent with Warm Restarts

By Ilya Loshchilov, Frank Hutter. 2016

10,851 citations recorded by the source corpus.

Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Quoc V. Le
  2. Mohit Bansal
  3. Yoshua Bengio
  4. Jianfeng Gao
  5. Sergey Levine
  6. Percy Liang
  7. Luke S. Zettlemoyer
  8. Dacheng Tao

In Computational Mechanics

  1. Francis R. Bach
  2. Peter Richtárik
  3. Zeyuan Allen-Zhu
  4. Julien Mairal
  5. Simon Lacoste-Julien
  6. Jitendra Malik
  7. Nicolas Loizou
  8. Michael I. Jordan

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Yi Yang
  8. Andrew Zisserman

In Optimization & Training

  1. Quoc V. Le
  2. Yoshua Bengio
  3. Frank Hutter
  4. Soham De
  5. Simon Lacoste-Julien
  6. Samuel L. Smith
  7. Martin Jaggi
  8. Peter Richtárik

Citations in the corpus