On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent

By Noah Golmant, Nikita Vemuri, Zi-Mei Yao, Vladimir Feinberg, A. Gholami, Kai Rothauge, Michael W. Mahoney, Joseph E. Gonzalez. 2018

82 citations recorded by the source corpus.

Increasing the mini-batch size for stochastic gradient descent offers significant opportunities to reduce wall-clock training time, but there are a variety of theoretical and systems challenges that impede the widespread success of this technique. We investigate these issues, with an emphasis on time to convergence and total computational cost, through an extensive empirical analysis of network training across several architectures and problem domains, including image classification, image segmentation, and language modeling. Although it is common practice to increase the batch size in order to fully exploit available computational resources, we find a substantially more nuanced picture. Our main finding is that across a wide range of network architectures and problem domains, increasing the batch size beyond a certain point yields no decrease in wall-clock time to convergence for \emph{either} train or test loss. This batch size is usually substantially below the capacity of current systems. We show that popular training strategies for large batch size optimization begin to fail before we can populate all available compute resources, and we show that the point at which these methods break down depends more on attributes like model architecture and data complexity than it does directly on the size of the dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “For instance, Devarakonda et al. (2017) and Lin et al. (2018) used linear learning rate scaling and Anonymous (2019) tried constant, square root, and linear learning rate scaling heuristics.”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “The importance of understanding the benefits of modern parallel hardware has motivated a lot of recent work on training neural networks with larger batch sizes [Goyal et al., 2017, Osawa et al., 2018, McCandlish et al., 2018, Shallue et al., 2018]. To date, the most comprehensive empirical study of ”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Furthermore certain real-world data sets, it is well known that larger minibatches also lead to poor generalization and gradient saturation that lead to significant performance gaps between the ideal and practical speed up [12, 22, 41, 20].”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “3 LOCAL SGD SCALES TO LARGER BATCH SIZES THAN MINI-BATCH SGD The empirical studies (Shallue et al., 2018; Golmant et al., 2018) reveal the regime of maximal data parallelism across different tasks and models, where the large-batch training would reach the limit and additional parallelism provides no”
On Large-Cohort Training for Federated Learning (2021, background): “This mirrors issues of diminishing returns in large-batch training [14, 19, 51, 64].”
Augment Your Batch: Improving Generalization Through Instance Repetition (2020, background): “However, several papers claimed that working with large batch results in accuracy degradation [21, 7].”
Accelerating Stochastic Training for Over-parametrized Learning (2018, result): “Since m∗1 ≤ L1/L, the linear scaling regime is limited, which is consistent with the empirical observation for neural networks in (Golmant et al., 2018).”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, background): “(We should, however, point out that there are many subtleties in practical large-batch training (Golmant et al., 2018; Shallue et al., 2018).”
Augment your batch: better training with larger batches (2019, background): “However, several papers claimed that working with large batch results in accuracy degradation (Masters & Luschi, 2018; Golmant et al., 2019).”