ImageNet Training in Minutes

By Yang You, Zhao Zhang, Cho-Jui Hsieh, J. Demmel, Kurt Keutzer. 2017

470 citations recorded by the source corpus.

In this paper, we investigate large scale computers' capability of speeding up deep neural networks (DNN) training. Our approach is to use large batch size, powered by the Layer-wise Adaptive Rate Scaling (LARS) algorithm, for efficient usage of massive computing resources. Our approach is generic, as we empirically evaluate the effectiveness on two neural networks: AlexNet and ResNet-50 trained with the ImageNet-1k dataset while preserving the state-of-the-art test accuracy. Compared to the baseline of a previous study from a group of researchers at Facebook, our approach shows higher test accuracy on batch sizes that are larger than 16K. Using 2,048 Intel Xeon Platinum 8160 processors, we reduce the 100-epoch AlexNet training time from hours to 11 minutes. With 2,048 Intel Xeon Phi 7250 Processors, we reduce the 90-epoch ResNet-50 training time from hours to 20 minutes. Our implementation is open source and has been released in the Intel distribution of Caffe v1.0.7.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Optimization & Training

Citations in the corpus

An Empirical Model of Large-Batch Training (2018, methodology): “For example, batch sizes of 8 thousand [GDG + 17], 16 thousand [SKYL17], 32 thousand [YGG17, YZH + 17, ASF17], and even 64 thousand [JSH + 18] examples have been effectively employed to train ImageNet, and batch sizes of thousands have been effective for language models and generative models…”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “These follow-on studies (Codreanu et al., 2017; You et al., 2017; Akiba et al., 2017) confirm that the best solution quality for a given batch size will also depend on the exact optimization techniques used.”
Beyond human-level accuracy: computational challenges in deep learning (2019, methodology): “Researchers have shown how to train image and language applications quickly using data-parallel scaling to reduce per-accelerator activation memory requirements [14, 26, 40].”
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021, methodology): “[47] both demonstrate the use of High Performance Computing techniques to train highly-accurate ImageNet models in minutes.”
Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Increasing interest has focused on large batch training (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), in an attempt to increase the step size and reduce the number of parameter updates required to train a model.”
MegDet: A Large Mini-Batch Object Detector (2017, methodology): “[39] presents a training setting which can finish the ResNet50 training in 31 minutes without losing classification accuracy.”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, methodology): “All the successful implementations are based on momentum SGD (He et al., 2016; Goyal et al., 2017) or L ARS optimizer (Ying et al., 2018; Jia et al., 2018; Mikami et al., 2018; You et al., 2018; Yamazaki et al., 2019).”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “At the same time, between 2009-2021, the community managed to “democratize” means to train SotA models in less than 11 minutes [64], made available pre-trained models that are as compressed as 3.8 MB (the MeliusNet22 model) and as fast as 17ms inference-time on a commercially available smartphone…”
On Layer Normalization in the Transformer Architecture (2020, background): “Goyal et al. (2017); He et al. (2019); You et al. (2018) showed that a learning rate warm-up stage is preferred when training neural networks with extremely large batch sizes.”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “Such a solution is equivalent to single-threaded SGD with large mini-batch sizes (Goyal et al., 2017; You et al., 2017a,b, 2019).”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, methodology): “Beyond them, a simple but powerful method is to reduce the communication frequency by allowing more local updates [58, 39, 22, 47, 44], which we focus on in this paper.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Recent schemes for scaling training to a large number of workers rely on standard mini-batch SGD (1) with very large overall batch sizes (Shallue et al., 2018; You et al., 2018; Goyal et al., 2017), i.”
Lookahead Optimizer: k steps forward, 1 step back (2019, background): “Large-scale distributed optimization algorithms [10, 45] have shown impressive performance in combination with improved learning rate scheduling schemes [42, 35], yet variants of SGD remain the core algorithm in the distributed systems.”
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, background): “…(Shamir et al., 2014; Jaggi et al., 2014; Ma et al., 2015; Reddi et al., 2016; Ma et al., 2017), reducing the communication frequency (McDonald et al., 2009; Zinkevich et al., 2010; You et al., 2017; Stich, 2018) or distributing along features (Richtárik & Takáč, 2016; Fercoq et al., 2014).”
Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes (2017, methodology): “[10] Xeon 8160 × 1600 Intel Caffe 16,000 31 min 75.”
Drawing early-bird tickets: Towards more efficient training of deep networks (2019, unknown): “Specifically, all the energy consumption of full-precision models are obtained by training the corresponding models in an embedded GPU (NVIDIA JETSON TX2).”
A Multigrid Method for Efficiently Training Video Models (2019, background): “, [8, 24, 33, 39]), pre-training [3, 11], distributed training [13, 50], or advances in hardware [22] and software design [4, 6].”