100-epoch ImageNet Training with AlexNet in 24 Minutes

By Yang You, Zhao Zhang, Cho-Jui Hsieh, J. Demmel. 2017

61 citations recorded by the source corpus.

Finishing 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU takes 14 days. This training requires 10^18 single precision operations in total. On the other hand, the world's current fastest supercomputer can finish 2 * 10^17 single precision operations per second (Dongarra et al 2017, https://www.top500.org/lists/2017/06/). If we can make full use of the supercomputer for DNN training, we should be able to finish the 90-epoch ResNet-50 training in five seconds. However, the current bottleneck for fast DNN training is in the algorithm level. Specifically, the current batch size (e.g. 512) is too small to make efficient use of many processors For large-scale DNN training, we focus on using large-batch data-parallelism synchronous SGD without losing accuracy in the fixed epochs. The LARS algorithm (You, Gitman, Ginsburg, 2017, arXiv:1708.03888) enables us to scale the batch size to extremely large case (e.g. 32K). We finish the 100-epoch ImageNet training with AlexNet in 24 minutes. Same as Facebook's result (Goyal et al 2017, arXiv:1706.02677), we finish the 90-epoch ImageNet training with ResNet-50 in one hour. Furthermore, when we increase the batch size to above 10K, our accuracy is much higher than Facebook's on corresponding batch sizes.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Don't Use Large Mini-Batches, Use Local SGD (2018, background): “We limit ResNet-50 training to 90 passes over the data in total, and the data is disjointly partitioned and is re-shuffled globally every epoch.”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “)) ▷ Compute forward evaluation 5: д mb ←∇ℓ(w mb, f ) ▷ Compute gradient using backpropagation 6: ∆w ←u(д mb,w(0,...,t),t) ▷ Weight update rule 7: w(t+1)←w mb + ∆w ▷ Store parameters 8: end for epochs[83,250]. As opposed to the original SGD, shuffle-based processing entails without-replacement sampl”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, methodology): “We observe markedly worse performance for these techniques in domains other than image classiﬁcation, where large batch optimization has received the most attention [15, 33].”
A System for Massively Parallel Hyperparameter Tuning (2018, background): “…degree of parallelism for au-toscaling a training task using an efﬁciency criteria motivated by the observation that speedups from parallel training do not scale linearly with the number of GPUs (Krizhevsky, 2014; Szegedy et al., 2014; You et al., 2017; You et al., 2017; Goyal et al., 2017).”
Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutes (2018, methodology): “Given the results reported in [38] testing the Intel ® Xeon Phi TM architecture on distributed RL training would also be an interesting experiment.”
The Secret Sharer: Measuring Unintended Neural Network Memorization & Extracting Secrets (2018, background): “, 32K) to train models many orders of magnitude more quickly than previously possible [23, 54, 55].”
Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks (2019, methodology): “[35]) combines LR warmup with Layer-wise Adaptive Rate Scaling (LARS) [34].”
Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train (2017, unknown): “As far as we are aware, besides the research from [3] and [5], most the other ResNet-50 scaling works report a validation accuracy lower than the 75.”
TicTac: Accelerating Distributed Deep Learning with Communication Scheduling (2018, background): “all reduce) (Goyal et al., 2017; Cho et al., 2017; Amodei et al., 2015; You et al., 2017; Akiba et al., 2017) or broadcast (Zhang et al.”