How to scale distributed deep learning?

By Peter H. Jin, Qiaochu Yuan, Forrest N. Iandola, Kurt Keutzer. 2016

138 citations recorded by the source corpus.

Training time on large datasets for deep neural networks is the principal workflow bottleneck in a number of important applications of deep learning, such as object classification and detection in automatic driver assistance systems (ADAS). To minimize training time, the training of a deep neural network must be scaled beyond a single machine to as many machines as possible by distributing the optimization method used for training. While a number of approaches have been proposed for distributed stochastic gradient descent (SGD), at the current time synchronous approaches to distributed SGD appear to be showing the greatest performance at large scale. Synchronous scaling of SGD suffers from the need to synchronize all processors on each gradient step and is not resilient in the face of failing or lagging processors. In asynchronous approaches using parameter servers, training is slowed by contention to the parameter server. In this paper we compare the convergence of synchronous and asynchronous SGD for training a modern ResNet network architecture on the ImageNet classification problem. We also propose an asynchronous method, gossiping SGD, that aims to retain the positive features of both systems by replacing the all-reduce collective operation of synchronous training with a gossip aggregation algorithm. We find, perhaps counterintuitively, that asynchronous SGD, including both elastic averaging and gossiping, converges faster at fewer nodes (up to about 32 nodes), whereas synchronous SGD scales better to more nodes (up to about 100 nodes).

Subfields: Artificial Intelligence, Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “For ﬁxed topology algorithms, the communication network is ﬁxed, for example, (Jin et al., 2016; Lian et al., 2017b; Shen et al., 2018; Tang et al., 2018b;c).”
Wireless Network Intelligence at the Edge (2018, methodology): “The communication among devices can be enabled by a push–pull gossip algorithm [104] or by following a predefined communication network topology [105], [106].”
Distributed Learning over Unreliable Networks (2018, background): “Specifically, Jin et al. (2016) proposes to scale the gradient aggregation process via a gossip-like mechanism.”
Stochastic Gradient Push for Distributed Deep Learning (2018, neutral): “ (quantized, infrequent, and inexact averaging) is an interesting direction for future work. Blot et al.(2016) report initial experimental results on smallscale experiments with an SGP-like algorithm.Jin et al. (2016) make a theoretical connection between PUSHSUMbased methods and Elastic Averaging S”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, methodology): “Recently, decentralized averaging was successfully applied to deep learning in (Jin et al., 2016; Jiang et al., 2017; Lian et al., 2017) which also provide convergence analyses for 1 local update per worker.”
Collaborative Deep Learning in Fixed Topology Networks (2017, background): “O(γ) No No No Gossip SGD [7] Str-con Lip.”
Demystifying Parallel and Distributed Deep Learning (2018, neutral): “ Analysis 1:23 Category Method Model Consistency Synchronization Synchronous[45,112,173,210,224,246] Stale-Synchronous[88,98,120,156,261] Asynchronous[55,56,127,182,190,205,260] Nondeterministic Comm.[54,121,201] Parameter Distribution and Communication Centralization Parameter Server (PS)[52,112,12”
On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent (2018, background): “With the hope of keeping training times manageable as dataset sizes escalate, recent work has focused on the development of techniques that allow practitioners to increase the batch size to make use of growing computational resources [16, 15, 32].”
100-epoch ImageNet Training with AlexNet in 24 Minutes (2017, background): “A variety of asynchronous approaches have been proposed (Recht et al. 2011) (Zhang, Choromanska, and LeCun 2015a) (Jin et al. 2016) (Mitliagkas et al. 2016).”
Deep Learning With Edge Computing: A Review (2019, background): “[132] proposed gossiping SGD based on their study of convergence rates for synchronous and asynchronous SGD.”
Integrated Model, Batch, and Domain Parallelism in Training Neural Networks (2017, background): “One possibility to break this barrier for parallel training is the family of asynchronous SGD methods [4, 7, 11, 18, 20, 28].”
The future of digital health with federated learning (2020, background): “The second characteristic of FL - to distribute the compute - has been well studied in recent years [49, 50, 51].”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, methodology): “Amiri and Gunduz [57] implement DSGD at the wireless edge with the help of a remote parameter server and further develop DSGD in digital and analog schemes, respectively.”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, background): “There are two main approaches for synchronizing gradients: synchronous [4], [19], [25], [33] and asynchronous [12], [26], [34].”
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks (2018, background): “Gossiping SGD [23], considered as a decentralized version of Elastic Averaging SGD, makes use of the average of local weights instead of a center variable, thus eliminating the need of a master server.”
GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent (2018, methodology): “nstrated the limitations of existing gossip based approaches, including the parameter server approaches, which can be considered as an extreme form of gossip. The approaches presented 4 by Jin et al. [41] and Blot et al. [33] which use random gossip suffer from communication imbalance, poor diffusio”