On the Generalization Benefit of Noise in Stochastic Gradient Descent

By Samuel L. Smith, Erich Elsen, Soham De. 2020

129 citations recorded by the source corpus.

It has long been argued that minibatch stochastic gradient descent can generalize better than large batch gradient descent in deep neural networks. However recent papers have questioned this claim, arguing that this effect is simply a consequence of suboptimal hyperparameter tuning or insufficient compute budgets when the batch size is large. In this paper, we perform carefully designed experiments and rigorous hyperparameter sweeps on a range of popular models, which verify that small or moderately large batch sizes can substantially outperform very large batches on the test set. This occurs even when both models are trained for the same number of iterations and large batches achieve smaller training losses. Our results confirm that the noise in stochastic gradients can enhance generalization. We study how the optimal learning rate schedule changes as the epoch budget grows, and we provide a theoretical account of our observations based on the stochastic differential equation perspective of SGD dynamics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Soham De
2. Quoc V. Le
3. Samuel L. Smith
4. Andrew Brock
5. Karen Simonyan
6. Razvan Pascanu
7. Stanislav Fort
8. Yoshua Bengio

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Soham De
4. Andrew Zisserman
5. Ming-Hsuan Yang
6. Trevor Darrell
7. Xiaogang Wang
8. Jitendra Malik

In Optimization & Training

1. Soham De
2. Andrew Brock
3. Samuel L. Smith
4. Karen Simonyan
5. Jamie Hayes
6. Leonard Berrada
7. Borja Balle
8. Razvan Pascanu

In Scaling & Learning Theory

1. Soham De
2. Samuel L. Smith
3. Andrew Brock
4. Razvan Pascanu
5. Stanislav Fort
6. Karen Simonyan

Citations in the corpus

• High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Although large-batch training does not achieve higher test accuracies within a fixed epoch budget (Smith et al., 2020), it does achieve a given test accuracy in fewer parameter updates, significantly improving training speed when parallelized across multiple devices (Goyal et al.”
• Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, result): “As noted in Smith et al. (2020), we also find that this relationship breaks down past batch size 1024 for our unnormalized ResNets, as opposed to 2048 or 4096 for normalized ResNets.”
• Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error (2021, background): “Note, by contrast, performance degrades for very large batch sizes in standard pipelines [Smith et al., 2020].”
• Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, methodology): “Similar scaling behaviour has previously been observed in non-private training (Ma et al., 2018; McCandlish et al., 2018; Smith et al., 2020), while linearly scaling the learning rate with the batch size was suggested as a heuristic for private training by Tramèr and Boneh (2021), although as we sho”