Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train

By Valeriu Codreanu, Damian Podareanu, Vikram A. Saletore. 2017

58 citations recorded by the source corpus.

For the past 5 years, the ILSVRC competition and the ImageNet dataset have attracted a lot of interest from the Computer Vision community, allowing for state-of-the-art accuracy to grow tremendously. This should be credited to the use of deep artificial neural network designs. As these became more complex, the storage, bandwidth, and compute requirements increased. This means that with a non-distributed approach, even when using the most high-density server available, the training process may take weeks, making it prohibitive. Furthermore, as datasets grow, the representation learning potential of deep networks grows as well by using more complex models. This synchronicity triggers a sharp increase in the computational requirements and motivates us to explore the scaling behaviour on petaflop scale supercomputers. In this paper we will describe the challenges and novel solutions needed in order to train ResNet-50 in this large scale environment. We demonstrate above 90\% scaling efficiency and a training time of 28 minutes using up to 104K x86 cores. This is supported by software tools from Intel's ecosystem. Moreover, we show that with regular 90 - 120 epoch train runs we can achieve a top-1 accuracy as high as 77\% for the unmodified ResNet-50 topology. We also introduce the novel Collapsed Ensemble (CE) technique that allows us to obtain a 77.5\% top-1 accuracy, similar to that of a ResNet-152, while training a unmodified ResNet-50 topology for the same fixed training budget. All ResNet-50 models as well as the scripts needed to replicate them will be posted shortly.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “These follow-on studies (Codreanu et al., 2017; You et al., 2017; Akiba et al., 2017) confirm that the best solution quality for a given batch size will also depend on the exact optimization techniques used.”
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes (2019, background): “…without degrading the performance, thereby, ﬁnishing R ES N ET - 50 training on ImageNet in a few minutes (You et al., 2018; Iandola et al., 2016; Codreanu et al., 2017; Akiba et al., 2017; Jia et al., 2018; Smith et al., 2017; Martens & Grosse, 2015; Devarakonda et al., 2017; Mikami et al.,…”
ImageNet-21K Pretraining for the Masses (2021, methodology): “We shall refer to the full dataset as ImageNet-21K, 21 following [27] (although other papers sometimes described it as ImageNet-22K [8]).”
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes (2018, methodology): “Similar work have also been demonstrated in [7][19].”
OD-SGD (2020, background): “As a matter of fact, a series of research has been done to increase the batch size [34, 35, 36, 37], and the most important thing in this ﬁeld is keeping the model accuracy as the batch size goes up.”
Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks (2019, methodology): “[1]) uses LR warmup and dynamic weight decay (WD).”
TF-Replicator: Distributed Machine Learning for Researchers (2019, background): “Subsequent studies (Codreanu et al., 2017; Smith et al., 2017; You et al., 2018) reduced this time to 15 minutes through a combination of larger batches and more GPUs (e.g. batch size 32K with 1024 Tesla P100s in (Akiba et al., 2017)), along with algorithmic tricks such as batch normalization and…”
SparCML: High-Performance Sparse Communication for Machine Learning (2018, background): “Due to their increased capacity, wide residual networks are popular when transferring to more complex tasks, such as ImageNet-10K and 22K [15].”
Reducing BERT Pre-Training Time from 3 Days to 76 Minutes (2019, methodology): “the BERT pre-training includes two stages: (1) the first 9/10 of the total epochs use a sequence length of 128 and (2) the last 1/10 of the total epochs use a sequence length of 512.”