Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study

By Suyog Gupta, Wei Zhang, Fei Wang. 2015

179 citations recorded by the source corpus.

Deep learning with a large number of parametersrequires distributed training, where model accuracy and runtimeare two important factors to be considered. However, there hasbeen no systematic study of the tradeoff between these two factorsduring the model training process. This paper presents Rudra, aparameter server based distributed computing framework tunedfor training large-scale deep neural networks. Using variants ofthe asynchronous stochastic gradient descent algorithm we studythe impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtimeperformance and model accuracy. We introduce a new learningrate modulation strategy to counter the effect of stale gradientsand propose a new synchronization protocol that can effectivelybound the staleness in gradients, improve runtime performanceand achieve good model accuracy. Our empirical investigationreveals a principled approach for distributed training of neuralnetworks: the mini-batch size per learner should be reducedas more learners are added to the system to preserve the modelaccuracy. We validate this approach using commonly-used imageclassification benchmarks: CIFAR10 and ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling (2019, background): “Only a few previous works explore the second factor from a theoretical perspective, see (Dutta et al., 2018; Gupta et al., 2016) — these works consider the centralized parameter server model.”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, background): “Synchronization delays can be alleviated via asynchronous gradient aggregation (Recht et al., 2011; Cui et al., 2014; Gupta et al., 2016; Mitliagkas et al., 2016; Dutta et al., 2018).”
Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD (2018, neutral): “K-sync SGD: This is a generalized form of synchronous SGD, also suggested in [Chen et al., 2016, Gupta et al., 2016] to offer some resilience to straggling as the PS does not wait for all the learners to finish.”
Asynchronous Decentralized Parallel Stochastic Gradient Descent (2017, methodology): “ilarly to AllReduce and converges better than D-PSGD when running with 16,32,64,128 workers. How to maintain convergence while increasing M 4n is an active ongoing research area Goyal et al. [2017 ], Zhang et al. 2016 ] and it is orthogonal to the topic of this paper. For 64 and 128 workers, we adop”
Collaborative Deep Learning in Fixed Topology Networks (2017, background): “Scaling up deep learning [1] is becoming increasingly crucial for large-scale applications where the sizes of both the available data as well as the models are massive [2].”
Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD (2018, methodology): “A popular distributed SGD implementation is the parameter server framework [7, 4, 14, 10, 18] where in each iteration, worker nodes compute gradients on one mini-batch of data and a central parameter server aggregates these gradients (synchronously or asynchronously) and updates the parameter vector”
Fast Algorithms for Convolutional Neural Networks (2015, background): “State of the art convnet architectures for image recognition use deep networks of 3 × 3 convolutional layers, because they achieve better accuracy with fewer weights than shallow networks with larger ﬁlters [11, 7].”
Demystifying Parallel and Distributed Deep Learning (2018, background): “Hierarchical parameter servers [88,253] (Fig.”
Training Deep Neural Networks with 8-bit Floating Point Numbers (2018, methodology): “lso follows the FP32 baseline. 3 Experimental Results Reduced-precision emulated experiments were performed using NVIDIA GPUs. The software platform is an in-house distributed deep learning framework [7]. The three GEMM computations share the same bit-precision and chunk-size: FP8 for input operands”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, background): “Training models effectively while maintaining consistency, fast convergence, and accuracy remains however challenging [441].”
GaDei: On Scale-Up Training as a Service for Deep Learning (2016, background): “B. Verification of GaDei’s communication protocol Due to the space limit, we prove GaDei is deadlock-free and verify GaDei’s liveness property in a companion arXiv paper[11].”
DeepChain: Auditable and Privacy-Preserving Deep Learning with Blockchain-Based Incentive (2019, background): “In order to achieve even higher accuracy, huge amount of data must be fed to deep learning models, incurring excessively high computational overhead [6], [7].”
Distributed Deep Learning for Question Answering (2015, unknown): “IBM’s Rudra system [7] is a master-client based distributed framework where the servers are organized as a tree structure to save communication overhead.”
Distributed deep learning on edge-devices: Feasibility via adaptive compression (2017, background): “The staleness, proportional to the number of workers (please refer to [9], [18] or [29] for in depth phenomenon explanation), is an important issue in asynchronous SGD in general.”
AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training (2017, methodology): “The software platform is an in-house distributed deep learning framework ((Gupta, Zhang, and Wang 2016), (Nair and Gupta 2017)).”