Scalable distributed DNN training using commodity GPU cloud computing

By Nikko Ström. 2015

591 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In ML Systems & Hardware

Citations in the corpus

Training Neural Networks with Fixed Sparse Masks (2021, background): “An orthogonal approach to reducing communication costs is to have workers only update a small subset of the model’s parameters [2, 13, 44, 45].”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “Various approaches have been proposed to reduce the communication overhead of distributed SGD, such as quantization (Seide et al., 2014; Strom, 2015; Wen et al., 2017; Alistarh et al., 2016; Bernstein et al., 2018; Karimireddy et al., 2019; Zheng et al., 2019) and sparsification (Aji and Heafield,…”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Gradient vector sparsification is another approach analyzed in [4, 40, 39, 5, 30, 34, 9, 3, 36, 21, 32].”
Decentralized Deep Learning with Arbitrary Communication Compression (2019, methodology): “Communication compression with quantization has been popularized in the deep learning community by the reported successes in (Seide et al., 2014; Strom, 2015).”
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations (2019, methodology): “, where nodes locally select Topk values of the gradient in absolute value and transmit these at full precision [5], [14], [16]–[19], while maintaining errors in local nodes for later compensation.”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “To reduce the communication cost at each iteration, compression or sparsification based parallel SGD are considered in (Seide et al., 2014; Strom, 2015; Aji & Heafield, 2017; Wen et al., 2017; Alistarh et al., 2017).”
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, background): “Strom [24] presents an approach (later modified by [26]) in which only gradients with a magnitude greater than a certain predefined threshold are sent to the server.”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “Interestingly, all the practical works (Seide et al., 2014; Strom, 2015; Lin et al., 2018) use some form of error-feedback.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Another promising line of research addressing the communication bottleneck of large scale training is to use quantization [31, 32, 33] or more aggressive sparsification [34, 35, 36] of gradients.”
Local SGD Converges Fast and Communicates Little (2018, background): “A very aggressive—and promising—sparsification method is to keep only very few coordinates of the stochastic gradient by considering only the coordinates with the largest magnitudes [3, 9, 18, 32, 35, 36, 37].”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, background): “Gradient Sparsiﬁcation Strom (2015) proposed threshold quantization to only send gradients larger than a predeﬁned constant threshold.”
QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks (2016, methodology): “This approach is shown to perform well on large-scale Amazon proprietary datasets by [33].”
Sparsified SGD with Memory (2018, methodology): “As the usefulness of SGD with sparsification techniques has already been shown in practical applications [1, 9, 21, 33, 37] we focus here on a few particular aspects.”
signSGD: compressed optimisation for non-convex problems (2018, positive): “As for existing gradient compression schemes, Seide et al. (2014) and Strom (2015) demonstrated empirically that 1-bit quantisation can still give good performance whilst dramatically reducing gradient communication costs in distributed systems.”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, background): “Other approaches for low-communication training focus on sparsification of gradients, either by thresholding small entries or by random sampling [6, 22, 23, 24, 25, 26, 27, 28].”
Sparse Communication for Distributed Gradient Descent (2017, methodology): “Strom (2015) proposed threshold quantiza-a tion, which only sends gradient updates that larger than a predeﬁned constant threshold.”
Collaborative Deep Learning in Fixed Topology Networks (2017, methodology): “Related work: Apart from the algorithms mentioned above, a few other related works exist, including a distributed system called Adam for large deep neural network (DNN) models [12] and a distributed methodology by Strom [13] for DNN training by controlling the rate of weight-update to reduce the amo”
Lessons from Building Acoustic Models with a Million Hours of Speech (2019, methodology): “We explored the Gradient Threshold Compression method (GTC) [35] and Blockwise Model-Update Filtering (BMUF) [41].”