AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

By Aditya Devarakonda, Maxim Naumov, Michael Garland. 2017

156 citations recorded by the source corpus.

Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a new training approach that, rather than statically choosing a single batch size for all epochs, adaptively increases the batch size during the training process. Our method delivers the convergence rate of small batch sizes while achieving performance similar to large batch sizes. We analyse our approach using the standard AlexNet, ResNet, and VGG networks operating on the popular CIFAR-10, CIFAR-100, and ImageNet datasets. Our results demonstrate that learning with adaptive batch sizes can improve performance by factors of up to 6.25 on 4 NVIDIA Tesla P100 GPUs while changing accuracy by less than 1% relative to training with fixed batch sizes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, background): “([DNG17] also suggested increasing learning rate and batch size together, but with different motivation.)”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Assuming an epoch budget when comparing solution quality between batch sizes (Masters and Luschi, 2018; Goyal et al., 2017; Lin et al., 2018; Devarakonda et al., 2017), in eﬀect, limits an investigation to the perfect scaling region of the steps to result vs batch size curve (see Figure 1).”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “Several recent works successfully scaled the batch size to large values using adaptive learning rates without degrading the performance, thereby, finishing RESNET50 training on ImageNet in a few minutes (You et al., 2018; Iandola et al., 2016; Codreanu et al., 2017; Akiba et al., 2017; Jia et al., 2”
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, background): “Theoretical upper bounds on the performance of low-precision SGD [9] and empirical observations of implemented low-precision algorithms [6] further confirm that current algorithms are limited by this precision-accuracy tradeoff.”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, background): “However, there is a growing body of theoretical and empirical results suggesting that large batch sizes adversely affect the generalization performance of the ﬁnal model [31, 17, 7].”
Reducing BERT Pre-Training Time from 3 Days to 76 Minutes (2019, background): “The trust ratio of LAMB is defined in Equation (3), which computes the denominator in the element-wise way rather than the layer-wise way. lamb_trust_ratio = |w| |g + weight_decay × w| (3)”
Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks (2018, background): “The use of small batch sizes to encourage rapid convergence in early epochs, and then progressively increasing the batch size is yet another successful approach [8, 24].”
Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs (2018, background): “The use of small batch sizes to encourage rapid convergence in early epochs, and then progressively increasing the batch size is yet another successful approach [8, 24].”
Parallelizing Word2Vec in Shared and Distributed Memory (2016, methodology): “As for future work, our plans include asynchronous model update similar to parameter sever [26], more efficient sub-model synchronization strategy as well as improving the rate of convergence by usingAdaBatch [27] and LARS [28].”
A Progressive Batching L-BFGS Method for Machine Learning (2018, unknown): “This approach was parallelized by Devarakonda et al. (2017).”
On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization (2019, result): “…networks, practitioners have observed that SGD using dynamically increasing batch sizes can converges to similar test accuracy with the same number of epochs but significantly fewer number of batches when compared with SGD with small batch sizes (Devarakonda et al., 2017; Smith et al., 2018).”