Train longer, generalize better: closing the generalization gap in large batch training of neural networks

By Elad Hoffer, Itay Hubara, Daniel Soudry. 2017

889 citations recorded by the source corpus.

Background: Deep learning models are typically trained using stochastic gradient descent or one of its variants. These methods update the weights using their gradient, estimated from a small fraction of the training data. It has been observed that when using large batch sizes there is a persistent degradation in generalization performance - known as the "generalization gap" phenomena. Identifying the origin of this gap and closing it had remained an open problem. Contributions: We examine the initial high learning rate training phase. We find that the weight distance from its initialization grows logarithmically with the number of weight updates. We therefore propose a "random walk on random landscape" statistical model which is known to exhibit similar "ultra-slow" diffusion behavior. Following this hypothesis we conducted experiments to show empirically that the "generalization gap" stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting the training regime used. We further investigate different techniques to train models in the large-batch regime and present a novel algorithm named "Ghost Batch Normalization" which enables significant decrease in the generalization gap without increasing the number of updates. To validate our findings we conduct several additional experiments on MNIST, CIFAR-10, CIFAR-100 and ImageNet. Finally, we reassess common practices and beliefs concerning training of deep models and suggest they may not be optimal to achieve good generalization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, background): “Some papers have reported a “generaliza-tion gap” in which large batch sizes lead to good training loss but cause a degradation in test loss, apparently unrelated to overﬁtting [KMN + 16, HHS17].”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “…rate for one batch size and then uses a heuristic to choose the learning rate for other batch sizes (Goyal et al., 2017; Keskar et al., 2017; Hoﬀer et al., 2017; Lin et al., 2018; Devarakonda et al., 2017; Anonymous, 2019) gives a systematic advantage to the batch size used in tuning (as…”
Scaling Neural Machine Translation (2018, background): “However, when looking at the number of weight updates (i.e. optimization steps) large batches converge faster (Hoffer et al., 2017).”
Scaling description of generalization with number of parameters in deep learning (2019, background): “Numerical and theoretical studies [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17] show that in the overparametrized regime, the loss landscape of DNNs is not rough with isolated minima as initially thought [18, 19], but instead has connected level sets and presents many flat directions, even near its g”
Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Increasing interest has focused on large batch training (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), in an attempt to increase the step size and reduce the number of parameter updates required to train a model.”
On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “We used batch normalization (Ioffe and Szegedy, 2015) with exponential moving average (EMA) decay of 0.997 for ResNet-32, and ghost batch normalization (Hoffer et al., 2017) with ghost batch size of 32 and EMA decay of 0 .”
Large Batch Training of Convolutional Networks (2017, methodology): “But using large batch may negatively impact the model accuracy, as was observed in Krizhevsky (2014), Li et al. (2014), Keskar et al. (2016), Hoffer et al. (2017),.. Increasing the global batch while keeping the same number of epochs means that you have fewer iterations to update weights.”
MegDet: A Large Mini-Batch Object Detector (2017, background): “g based on ResNet50 can be ﬁnished in one hour. [39] presents a training setting which can ﬁnish the ResNet50 training in 31 minutes without losing classiﬁcation accuracy. Besides the training speed, [17] investigates the generalization gap between large mini-batch and small minibatch, and propose t”
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, neutral): “To keep the same covariance, (Bottou et al., 2016; Li, 2017; Hoffer et al., 2017) choose the step size η ∼ √ m for batch size m.”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, positive): “This experiment provides additional evidence for prior work which has argued that using a smaller batch size yields more generalizable models [19]; however we ensure that all models reach the same final accuracy.”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “[21] Elad Hoffer, Itay Hubara, and Daniel Soudry, “Train longer, generalize better: closing the generalization gap”
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian (2019, background): “Generalization properties of stochastic gradient descent is extensively studied empirically [34, 28, 50, 16, 29, 24, 25].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Consistent with this perspective, the test accuracy of batch-normalized networks can often be improved by tuning the batch size, or by using ghost batch normalization in distributed training (Hoffer et al., 2017).”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “It was observed that training with extremely large batch was difﬁcult (Keskar et al., 2016; Hoffer et al., 2017).”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “It has been shown that training with large batch size (e.g. batch size > 103 for the case of ImageNet) typically degrades the performance both in terms of training and test error (Goyal et al., 2017; Chen & Huo, 2016; Hoffer et al., 2017; Keskar et al., 2016; Li, 2017; Li et al., 2014).”
Visualizing the Loss Landscape of Neural Nets (2017, background): “[23] arguing that generalization is not directly related to the curvature of loss surfaces, and some authors proposing specialized training methods that achieve good performance with large batch sizes [19, 14, 6].”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “[48] have presented an approach to find good network architecture using random search combined with welltrained set of shared weights.”
Random Search and Reproducibility for Neural Architecture Search (2019, neutral): “g ones. However, their work required several modifications to stabilize training (e.g., a tunable path dropout schedule over edges of the search DAG and a specialized ghost batch normalization scheme [19]). Furthermore, they only report experimental results on the CIFAR-10 benchmark, on which they f”
Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Properties (2020, background): “Smaller batch sizes have been shown to improve generalization [25, 26, 27] in some cases.”