Large Batch Training of Convolutional Networks

By Yang You, Igor Gitman, Boris Ginsburg. 2017

967 citations recorded by the source corpus.

A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Quoc V. Le
  2. Yoshua Bengio
  3. Mohit Bansal
  4. Jianfeng Gao
  5. Percy Liang
  6. Ruslan Salakhutdinov
  7. Dacheng Tao
  8. Sergey Levine

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Andrea Vedaldi

In Optimization & Training

  1. Peter Richtárik
  2. Tengyu Ma
  3. Martin Jaggi
  4. Yuanzhi Li
  5. Quoc V. Le
  6. Sebastian U. Stich
  7. Jason D. Lee
  8. Zeyuan Allen-Zhu

In Scaling & Learning Theory

  1. Yuanzhi Li
  2. Jason D. Lee
  3. Tengyu Ma
  4. Colin Wei
  5. Zeyuan Allen-Zhu
  6. Daniel Soudry
  7. Michael W. Mahoney
  8. Simon Shaolei Du

Citations in the corpus