Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback

By Shuai Zheng, Ziyue Huang, James T. Kwok. 2019

121 citations recorded by the source corpus.

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote achieves a 32x reduction on communication cost. However, its convergence is based on unrealistic assumptions and can diverge in practice. In this paper, we propose a general distributed compressed SGD with Nesterov's momentum. We consider two-way compression, which compresses the gradients both to and from workers. Convergence analysis on nonconvex problems for general gradient compressors is provided. By partitioning the gradient into blocks, a blockwise compressor is introduced such that each gradient block is compressed and transmitted in 1-bit format with a scaling factor, leading to a nearly 32x reduction on communication. Experimental results show that the proposed method converges as fast as full-precision distributed momentum SGD and achieves the same testing accuracy. In particular, on distributed ResNet training with 7 workers on the ImageNet, the proposed algorithm achieves the same testing accuracy as momentum SGD using full-precision gradients, but with $46\%$ less wall clock time.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “…communication overhead of distributed SGD, such as quantization (Seide et al., 2014; Strom, 2015; Wen et al., 2017; Alistarh et al., 2016; Bernstein et al., 2018; Karimireddy et al., 2019; Zheng et al., 2019) and sparsification (Aji and Heafield, 2017; Stich et al., 2018; Jiang and Agrawal, 2018).”
Natural Compression for Distributed Deep Learning (2019, background): “several state-of-the-art compressors (Karimireddy et al., 2019; Zheng et al., 2019; Bernstein et al., 2018; Vogels et al., 2019) by setting K so that the transmission amount are the same.”
FetchSGD: Communication-Efficient Federated Learning with Sketching (2020, unknown): “…compressed SGD converges when using a biased compression operator, existing methods ﬁrst show that their compression operator obeys a contraction property, and then they appeal to Stich et al. (2018) for convergence guarantees (Karimireddy et al., 2019b; Zheng et al., 2019; Ivkin et al., 2019a).”
High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning (2019, background): “Perhaps, the work most relevant to the current one is [9], where the whole stochastic gradient is cut into blocks.”
Efficient sparse collective communication and its application to accelerate distributed deep learning (2020, unknown): “While a new theoretical analysis for block-based sparsification is out of scope, we show that Block Random-k and Block Top-k are δ-compressors [30], and hence converge according to the ErrorFeedback theory [62, 71].”
High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning (2020, background): “improved by dividing the large-number of one-bit coefficients of a quantized gradient into blocks and scaling each block by the norm of the unquantized counterpart [12].”
CSER: Communication-efficient SGD with Error Reset (2020, methodology): “Very recently, Zheng et al. (2019) incorporates Nesterov’s momentum into EF-SGD with bidirectional communication and obtains faster convergence.”