On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

By Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, M. Smelyanskiy, Ping Tak Peter Tang. 2016

3,531 citations recorded by the source corpus.

The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.

Subfields: Artificial Intelligence, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Mohit Bansal
  3. Quoc V. Le
  4. Percy Liang
  5. Jianfeng Gao
  6. Sergey Levine
  7. Ruslan Salakhutdinov
  8. Luke S. Zettlemoyer

In Optimization & Training

  1. Quoc V. Le
  2. Yoshua Bengio
  3. Peter Richtárik
  4. Sebastian U. Stich
  5. Martin Jaggi
  6. Simon Lacoste-Julien
  7. Frank Hutter
  8. Tengyu Ma

In Scaling & Learning Theory

  1. Tengyu Ma
  2. Yuanzhi Li
  3. Jason D. Lee
  4. Behnam Neyshabur
  5. Colin Wei
  6. Greg Yang
  7. Samet Oymak
  8. J. Kolter

Citations in the corpus