Measuring the Effects of Data Parallelism on Neural Network Training

By Christopher J. Shallue, Jaehoon Lee, Joseph M. Antognini, Jascha Narain Sohl-Dickstein, Roy Frostig, George E. Dahl. 2018

484 citations recorded by the source corpus.

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “However, training with a 4× larger batch size can yield a different outcome than training for 4× as many steps [Shallue et al., 2018].”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, methodology): “CIFAR10 45,000 ResNet8 without BN Same as Shallue et al. [2018]. Constant ResNet32 with BN Ghost batch norm is used. Linear Decay VGG11 with BN Ghost batch norm is used. Linear Decay LM1B ⇠30M Two-layer Transformer Shallow model in Shallue et al. [2018] Constant”
Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “Nevertheless, batching is not a panacea: it puts pressure on the limited HBM memory capacity, and very large batch sizes can slow model convergence rates (Shallue et al., 2018; You et al., 2017; Lanchantin et al., 2020; Anil et al., 2021).”
ZeRO-Offload: Democratizing Billion-Scale Model Training (2021, methodology): “In the DL community, data parallelism is generally used as the de facto standard to scale DL training to multiple GPUs [5, 26, 35].”
On Empirical Comparisons of Optimizers for Deep Learning (2019, methodology): “For Fashion MNIST, CIFAR-10, ImageNet, and LM1B, our setup was identical to Shallue et al. (2019) except for the image pre-processing details described below.”
Reducing Noise in GAN Training with Variance Reduced Extragradient (2019, background): “In addition, two recent large scale studies showed that increased batch size has: (i) only marginal impact on single objective training (Shallue et al., 2018) and (ii) a surprisingly large performance improvement on GAN training (Brock et al.”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “Thus, several prior works carefully hand-tune training hyper-parameters, like learning rate and momentum, to avoid degradation of generalization performance (Goyal et al., 2017; Li, 2017; You et al., 2018; Shallue et al., 2018).”
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification (2019, methodology): “In Figure 6, we plot the accuracy against the effective learning rate deﬁned as η eﬀ = η/ (1 − β ) [Shallue et al., 2018] which suggests an optimal η eﬀ for each set of learning conditions.”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, background): “While P OWER SGD enables faster training with larger batch sizes, increasing batch sizes are known to eventually suffer from a ‘generalization gap’ (Shallue et al., 2018).”
Don't Use Large Mini-Batches, Use Local SGD (2018, methodology): “LOCAL SGD SCALES TO LARGER BATCH THAN MINI-BATCH SGD The empirical studies (Shallue et al., 2018; Defazio & Bottou, 2018) reveal the regime of maximal data parallelism across different tasks and models, where the large-batch training would reach the limit and additional parallelism provides no benef”
Demystifying Parallel and Distributed Deep Learning (2018, background): “[220] for an in-depth study of the effects of data parallelism on training.”
Hierarchical Federated Learning ACROSS Heterogeneous Cellular Networks (2019, background): “In [32], it has been shown that learning speed increases with the batchsize until certain point, hence we believe that a good strategy for federated learning is to adjust the batchsize according to the number users.”
Energy Efficient Federated Learning Over Wireless Communication Networks (2019, background): “7 shows the completion time versus the maximum average transmit power with different batch sizes [40], [41].”
Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs (2018, background): “This generalization gap caused by large mini-batch sizes have been studied extensively for various models and datasets [23].”
Federated Learning with Buffered Asynchronous Aggregation (2021, background): “This is similar to observations in conventional SGD training, where increasing the batch size eventually gives diminishing returns (Goyal et al., 2017; Shallue et al., 2018; Ott et al., 2018; You et al., 2017; 2018; 2019).”
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, background): “Previously Proposed Scaling Rules of Hyperparameters [11, 23, 31, 33] investigated the right way to scale learning rate with batch size, with sometimes conflicting proposals. which we summarize in Appendix F. [26] studied how learning rate (and batch size) should scale with width for MLPs and CNNs t”