SparCML: High-Performance Sparse Communication for Machine Learning

By Cédric Renggli, Dan Alistarh, Torsten Hoefler. 2018

149 citations recorded by the source corpus.

Applying machine learning techniques to the quickly growing data in science and industry requires highly-scalable algorithms. Large datasets are most commonly processed “data parallel” distributed across many nodes. Each node's contribution to the overall gradient is summed using a global allreduce. This allreduce is the single communication and thus scalability bottleneck for most machine learning workloads. We observe that frequently, many gradient values are (close to) zero, leading to sparse of sparsifyable communications. To exploit this insight, we analyze, design, and implement a set of communication-efficient protocols for sparse input data, in conjunction with efficient machine learning algorithms which can leverage these primitives. Our communication protocols generalize standard collective operations, by allowing processes to contribute arbitrary sparse input data vectors. Our generic communication library, Sparcml11Stands for Sparse Communication layer for Machine Learning, to be read as sparse ML., extends MPI to support additional features, such as non-blocking (asynchronous) operations and low-precision data representations. As such, Sparcml and its techniques will form the basis of future highly-scalable machine learning frameworks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, methodology): “For example, the parameter server [Abadi et al., 2016, Li et al., 2014], AllReduce [Renggli et al., 2018, Seide and Agarwal, 2016], adaptive distributed learning, diﬀerentially private distributed learning, distributed proximal primal-dual algorithm, non-smooth distributed optimization,…”
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “, 2014) and AllReduce that averages models over a ring topology (Renggli et al., 2018; Seide & Agarwal, 2016).”
Distributed Learning over Unreliable Networks (2018, methodology): “, 2016)1, as well as standard implementations of the AllReduce averaging operation in a decentralized setting (Seide and Agarwal, 2016; Renggli et al., 2018).”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, background): “Other approaches for low-communication training focus on sparsification of gradients, either by thresholding small entries or by random sampling [6, 22, 23, 24, 25, 26, 27, 28].”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “: Hyper-parameter Selection in Deep Neural Networks Using Parallel Particle Swarm Optimization, GECCO’17 [4] H.”
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection (2020, methodology): “Note that in DCD-PSGD, if c is larger than 4, it would lose much accuracy; and if c is set to 100 or 1000 like S-FegAvg or TopK-PSGD, it would not converge at all.”
Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, unknown): “2) Two SparCML [55] methods – SSAR_Split_allgather and DSAR_Split_allgather – that dominate performance for all SparCMLmethods in our experiments.”
HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks (2019, background): “Furthermore, one could explore to make training more efficient by incorporating Hessian to adjust precision throughout training [24], or use it to compress communication in distributed training [26, 12].”