Revisiting Small Batch Training for Deep Neural Networks

By Dominic Masters, Carlo Luschi. 2018

767 citations recorded by the source corpus.

Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footprint, which might also be exploited to improve machine throughput. In this paper, we review common assumptions on learning rate scaling and training duration, as a basis for an experimental comparison of test performance for different mini-batch sizes. We adopt a learning rate that corresponds to a constant average weight update per gradient calculation (i.e., per unit cost of computation), and point out that this results in a variance of the weight updates that increases linearly with the mini-batch size $m$. The collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet datasets show that increasing the mini-batch size progressively reduces the range of learning rates that provide stable convergence and acceptable test performance. On the other hand, small mini-batch sizes provide more up-to-date gradient calculations, which yields more stable and reliable training. The best performance has been consistently obtained for mini-batch sizes between $m = 2$ and $m = 32$, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “For example, Masters and Luschi (2018) found that test performance of ResNet-8 (without batch normalization) on CIFAR-10 with a ﬁxed epoch budget degraded after batch size 16, but considered only plain mini-batch SGD.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “…> 10 3 on ImageNet) typically degrades the performance both in terms of training and test error (often denoted as generalization gap ) (Chen & Huo, 2016; Li, 2017; Li et al., 2014; Keskar et al., 2017; Shallue et al., 2018; McCandlish et al., 2018; Golmant et al., 2018; Masters & Luschi, 2018).”
Differentially-Private "Draw and Discard" Machine Learning (2018, background): “The intuition for this is quite obvious: the convex nature of GLMs is more suitable for many small step updates in approximate directions than a much smaller number of steps in precise directions [35].”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “An opinion exists that for enhanced accuracy, mini-batch size and LR could be re-optimized after other hyper-parameters are fixed (Masters and Luschi, 2018).”
Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Properties (2020, background): “Smaller batch sizes have been shown to improve generalization [25, 26, 27] in some cases.”
Understanding the Role of Training Regimes in Continual Learning (2020, background): “[39], several other papers studied the correlation between minima wideness and generalization [32, 51, 33].”
Gandiva: Introspective Cluster Scheduling for Deep Learning (2018, background): “Some jobs can inherently result in low GPU utilization due to the use of small models [29] and/or the use of small batch sizes for better generalization [35].”
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark (2018, background): “…side, large minibatch training [31,38] and reduced precision [20,23,48] can help run iterations of the optimization algorithm faster and speed up “proxy” metrics such as time to process an epoch (“time-per-epoch”), but can prevent models from reaching the same accuracy on unseen data [24, 46, 47].”
The Break-Even Point on Optimization Trajectories of Deep Neural Networks (2020, methodology): “Our theoretical model is motivated by recent work on learning dynamics of neural networks (Goodfellow et al., 2014; Masters & Luschi, 2018; Wu et al., 2018; Yao et al., 2018; Xing et al., 2018; Jastrzebski et al., 2018; Lan et al., 2019).”
Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction (2019, background): “Recently [25] shows that small mini-batch sizes provide more up-to-date gradient calculations and yields more stable and reliable training.”
Scaling Distributed Machine Learning with In-Network Aggregation (2019, unknown): “These aggregations need to be performed frequently, as increasing the mini-batch size hurts convergence [61].”
Learning from One Continuous Video Stream (2023, background): “Note that while most approaches employ large batches, SGD with mini-batch size 1 is well known to work well [26] and is even said to generalize better than using large batches.”
The Effect of Network Width on the Performance of Large-batch Training (2018, background): “At the same time, several studies suggest that large-batch training can affect the generalizability of the models [16], can slow down convergence [17, 18, 19], and is more sensitive to hyperparameter mis-tuning [20].”
The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset (2020, methodology): “Our results agree with the ones obtained by Masters and Luschi [13], where the authors stated that smaller batch sizes should be used.”
PipeDream: generalized pipeline parallelism for DNN training (2019, background): “Furthermore, this strategy could prevent the model from reaching the desired target accuracy, since gradients are averaged over all training samples [10, 39].”
Gradient Compression Supercharged High-Performance Data Parallel DNN Training (2021, methodology): “We set batch sizes across different models by following literature [19, 33, 44, 69], instead of setting them to the largest value that a single GPU can sustain, since a too large batch size may lead to convergence problems [40, 61].”