Averaging Weights Leads to Wider Optima and Better Generalization

By Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, Andrew Gordon Wilson. 2018

2,084 citations recorded by the source corpus.

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Quoc V. Le
  3. Jianfeng Gao
  4. Yoshua Bengio
  5. Percy Liang
  6. Sergey Levine
  7. Luke S. Zettlemoyer
  8. Caiming Xiong

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Yi Yang
  8. Bernt Schiele

In Optimization & Training

  1. Quoc V. Le
  2. Yoshua Bengio
  3. Frank Hutter
  4. Richard Socher
  5. Alan L. Yuille
  6. Mohit Bansal
  7. Caiming Xiong
  8. Andrew Gordon Wilson

In Scaling & Learning Theory

  1. Andrew Gordon Wilson
  2. Pavel Izmailov
  3. Stanislav Fort
  4. Neil Houlsby
  5. Xiaohua Zhai
  6. Ari S. Morcos
  7. Lucas Beyer
  8. Sven Gowal

Citations in the corpus