On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
By Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, M. Smelyanskiy, Ping Tak Peter Tang. 2016
3,531 citations recorded by the source corpus.
The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.
Subfields: Artificial Intelligence, Optimization & Training, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Percy Liang
- Jianfeng Gao
- Sergey Levine
- Ruslan Salakhutdinov
- Luke S. Zettlemoyer
In Optimization & Training
- Quoc V. Le
- Yoshua Bengio
- Peter Richtárik
- Sebastian U. Stich
- Martin Jaggi
- Simon Lacoste-Julien
- Frank Hutter
- Tengyu Ma
In Scaling & Learning Theory
Citations in the corpus
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, background): “However, large batch training introduces complications into the optimization process that can result in reduced accuracy or longer time to convergence, offsetting the benefit of increased training throughput (Keskar et al., 2017).”
- An Empirical Model of Large-Batch Training (2018, background): “But noise may be useful early in training [NVL + 15, YPL + 17], perhaps because it leads to minima that generalize better [KMN + 16].”
- Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “The term “large batch” is inherently ambiguous, and in this case accompanies experiments in Keskar et al. (2017) that only compare two absolute batch sizes per data set, rather than charting out a curve to its apparent extremes.”
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density (2019, background): “Examples of such work, where some surrogate is defined for the curvature, include the debate on flat vs sharp minima [16, 5, 29, 15], explanations of the efficacy of residual connections [19] and batch normalization [25], the construction of low-energy paths between different local minima [6],…”
- Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “However they state that their analysis holds only in the neighborhood of a minimum, while Keskar et al. (2016) showed that the beneficial effects of noise are most pronounced at the start of training.”
- Large Batch Training of Convolutional Networks (2017, methodology): “Using the "linear LR scaling" Krizhevsky (2014) et al. (2016).”
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017, neutral): “Our comprehensive experiments in §5 show that optimization difficulty is the main issue with large minibatches, rather than poor generalization (at least on ImageNet), in contrast to some recent studies [20].”
- Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “…et al., 2011; Dozat, 2016; Martens & Grosse, 2015)) has become an important design choice, though understanding of its relationship to model generalization remains nascent (Shirish Keskar et al., 2016; Wilson et al., 2017; Shirish Keskar & Socher, 2017; Agarwal et al., 2020; Jacot et al., 2018).”
- A Closer Look at Memorization in Deep Networks (2017, background): “…proposed before, including looking at generalization error, trajectory length evolution (Raghu et al., 2016), analyzing Jacobians associated to different layers (Wang; Saxe et al., 2013), or the shape of the loss minima found by SGD (Im et al., 2016; Chaudhari et al., 2016; Keskar et al., 2016).”
- The large learning rate phase of deep learning: the catapult mechanism (2020, background): “Building on the observed correlation between lower curvature and generalization performance (Keskar et al., 2016; Jiang et al., 2020), we conjecture that optimal performance occurs in the large learning rate (catapult) phase, where the loss converges to a flatter minimum.”
- The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, positive): “significantly more memorization, as shown in Table 2. This experiment provides additional evidence for prior work which has argued that using a smaller batch size yields models which generalize better [26, 28, 32]. While this does give a method of reducing memorization for some models, it unfortunate”
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets (2022, methodology): “We then used the method described in Keskar et al. (2016) to calculate the sharpness approximation value, φ .”
- Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “Generalization properties of stochastic gradient descent is extensively studied empirically [34, 28, 50, 16, 29, 24, 25].”
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data (2018, background): “The training objectives of overparameterized networks in principle have many (approximate) global optima and some generalize better than the others [14, 8, 2], while empirical observations imply that the optimization process in practice prefers those with better generalization.”
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017, methodology): “Keskar et al. (2017) focused on a long observed phenomenon (LeCun et al.”
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “It was observed that training with extremely large batch was difficult (Keskar et al., 2016; Hoffer et al., 2017).”
- Shake-Shake regularization (2017, background): “Similarly, Stochastic Gradient Descent (SGD) (Bottou, 1998; Sutskever et al., 2013) can also be interpreted as Gradient Descent using noisy gradients and the generalization performance of neural networks often depends on the size of the mini-batch (see Keskar et al. (2017)).”
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks (2017, neutral): “And this looks in line with the observations in Keskar et al. [2016], in that, it appears that the LB solution and SB solutions are separated by a barrier, and that the latter of which generalizes better.”