Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

By Takuya Akiba, Shuji Suzuki, Keisuke Fukuda. 2017

320 citations recorded by the source corpus.

We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, methodology): “For example, batch sizes of 8 thousand [GDG + 17], 16 thousand [SKYL17], 32 thousand [YGG17, YZH + 17, ASF17], and even 64 thousand [JSH + 18] examples have been effectively employed to train ImageNet, and batch sizes of thousands have been effective for language models and generative models…”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “These follow-on studies (Codreanu et al., 2017; You et al., 2017; Akiba et al., 2017) confirm that the best solution quality for a given batch size will also depend on the exact optimization techniques used.”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “…performance, thereby, ﬁnishing R ES N ET - 50 training on ImageNet in a few minutes (You et al., 2018; Iandola et al., 2016; Codreanu et al., 2017; Akiba et al., 2017; Jia et al., 2018; Smith et al., 2017; Martens & Grosse, 2015; Devarakonda et al., 2017; Mikami et al., 2018; Osawa et al., 2018;…”
Stochastic Gradient Push for Distributed Deep Learning (2018, methodology): “Worker nodes compute local mini-batch gradients of the loss function on different subsets of the data, and then calculate an exact inter-node average gradient using either the ALLREDUCE communication primitive, in synchronous implementations (Goyal et al., 2017; Akiba et al., 2017), or using a centr”
signSGD: compressed optimisation for non-convex problems (2018, background): “Remark 1: switching optimisers after a warmup period is in fact commonly done by practitioners (Akiba et al., 2017).”
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, methodology): “61 times faster and higher top-1 test accuracy than the work in [2].”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, unknown): “A handful of works (Goyal et al., 2017; Cho et al., 2017; You et al., 2018; Akiba et al., 2017; Jia et al., 2018; Gupta et al., 2015) focus on reducing the total training time in paralleled, communication-efficient distributed settings.”
Reducing BERT Pre-Training Time from 3 Days to 76 Minutes (2019, methodology): “the BERT pre-training includes two stages: (1) the first 9/10 of the total epochs use a sequence length of 128 and (2) the last 1/10 of the total epochs use a sequence length of 512.”
SparCML: High-Performance Sparse Communication for Machine Learning (2018, methodology): “A second lossless method is executing extremely large batches, thus hiding the cost of communication behind larger computation [3, 20, 57, 58].”
TicTac: Accelerating Distributed Deep Learning with Communication Scheduling (2018, background): “all reduce) (Goyal et al., 2017; Cho et al., 2017; Amodei et al., 2015; You et al., 2017; Akiba et al., 2017) or broadcast (Zhang et al.”
Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks (2018, methodology): “[3] Tesla P100 × 1024 Chainer 32,768 RMSprop/SGD 3,519 15 min 74.”
GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent (2018, methodology): “Though this work did not focus on hyperparemter tuning, we expect GossipGraD would also benefit from hyperparameter tuning such as the proposed LARS method [65, 66], RMSprop warm-up [67], and slow-start learning rate schedules [68, 69].”
OD-SGD (2020, background): “As a matter of fact, a series of research has been done to increase the batch size [34, 35, 36, 37], and the most important thing in this ﬁeld is keeping the model accuracy as the batch size goes up.”
Fast Deep Neural Network Training on Distributed Systems and Cloud TPUs (2019, background): “[15] reported finishing the 90-epoch ResNet50 training within 15 minutes on 1,024 Nvidia P100 GPUs.”
Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs (2018, methodology): “[3] Tesla P100 × 1024 Chainer 32,768 RMSprop→ SGD 90 15 min 74.”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, background): “There are two main approaches for synchronizing gradients: synchronous [4], [19], [25], [33] and asynchronous [12], [26], [34].”
Deep Learning for Multigrade Brain Tumor Classification in Smart Healthcare Systems: A Prospective Survey (2020, methodology): “The CNNs used for the comparison are AlexNet [107], GoogleNet [105], VGG [70], SqueezeNet [108], MobileNet [109], and ResNet [110].”
Large Scale Language Modeling: Converging on 40GB of Text in Four Hours (2018, background): “Several recent publications seek to address long training times by leveraging distributed data parallelism and increasing the effective batch size during training [1], [17]– [20], taking advantage of advances in distributed deep learning and improvements in the memory size and compute capability of ”
ImageNet Training in Minutes (2017, background): “[1] reported finishing the 90-epoch ResNet-50 training within 15 minutes on 1,024 Nvidia P100 GPUs.”