QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding

By Dan Alistarh, Demjan Grubic, Jungshian Li, Ryota Tomioka, Milan Vojnovic. 2016

1,728 citations recorded by the source corpus.

Subfields: Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “In addition, signiﬁcant research has gone into algorithmic approaches to efﬁciently scale training [61], [62] by reducing communication cost via compression [63], [64], pipelining [65], and sharding [66], [67].”
A Unified Theory of Decentralized SGD with Changing Topology and Local Updates (2020, methodology): “In this work we do not consider gradient compression techniques (Alistarh et al., 2017; Stich et al., 2018; Tang et al., 2018a; 2019; Stich & Karimireddy, 2019)—but such orthogonal techniques could be added on top of our scheme—and instead only focus on local updates steps which are often efficient ”
Distributed Fixed Point Methods with Compressed Iterates (2019, methodology): “Usually, the exchanged information is usually some compressed gradient [1,7,27] or compressed model update [4,25] in a distributed master-worker setting.”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “Various approaches have been proposed to reduce the communication overhead of distributed SGD, such as quantization (Seide et al., 2014; Strom, 2015; Wen et al., 2017; Alistarh et al., 2016; Bernstein et al., 2018; Karimireddy et al., 2019; Zheng et al., 2019) and sparsification (Aji and Heafield, 2”
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, background): “Example 1 (Low-precision quantizer [Alistarh et al., 2017]).”
Robust and Communication-Efficient Collaborative Learning (2019, methodology): “In particular, efﬁcient gradient-compression methods are provided in (Alistarh et al., 2017; Seide et al., 2014; Bernstein et al., 2018) and deployed in the distributed master-worker setting.”
Decentralized Deep Learning with Arbitrary Communication Compression (2019, methodology): “• gsgd b (Alistarh et al., 2017).”
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, background): “A general theoretical analysis of centralized compressed parallel SGD can be found in Alistarh et al. [2017].”
QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, neutral): “While previous work [2, 4, 9, 24, 30, 36, 46, 56] had already introduced the idea of functions Qw ,Qa to quantize weights and/or activations in the forward pass, they required the weights W or gradients G, e to be represented in floating-point in the backward pass, in order to optimize accurately.”
Natural Compression for Distributed Deep Learning (2019, methodology): “, 2018), QSGD (Alistarh et al., 2017) and TernGrad (Wen et al.”
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “, 2016; Seide & Agarwal, 2016), and there are many methods being studied to improve the performance of distributed learning, such as communication efficient learning (Alistarh et al., 2017; Bernstein et al., 2018a; Seide et al., 2014), decentralized learning (He et al.”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “To reduce the communication cost at each iteration, compression or sparsification based parallel SGD are considered in (Seide et al., 2014; Strom, 2015; Aji & Heafield, 2017; Wen et al., 2017; Alistarh et al., 2017).”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, background): “…the coordinates of the gradient with only their sign (Seide et al., 2014; Carlson et al., 2015; Bernstein et al., 2018, 2019; Karimireddy et al., 2019), quantizing the individual coordinates (Alistarh et al., 2017; Wen et al., 2017), and low-rank approximation of the gradient (Wang et al., 2018).”
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication (2019, background): “A class of very common quantization operators is based on random dithering (Goodall, 1951; Roberts, 1962) that is in addition also unbiased, E ξ Q ( x ) = x , ∀ x ∈ R d , see (Alistarh et al., 2017; Wen et al., 2017; Zhang et al., 2017).”
Stochastic Gradient Push for Distributed Deep Learning (2018, background): “Related Work A variety of approaches have been proposed to accelerate distributed training of DNNs in the communcation bound setting, including quantizing gradients (Alistarh et al., 2007; Wen et al., 2007; Jia et al., 2018) and performing multiple local SGD steps at each node before averaging (McMa”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Another promising line of research addressing the communication bottleneck of large scale training is to use quantization [31, 32, 33] or more aggressive sparsification [34, 35, 36] of gradients.”
Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning (2018, methodology): “In Theorem 1 and Corollary 1, we follow the convention in literature (Ghadimi and Lan 2013) (Lian et al. 2017) (Alistarh et al. 2017) to use the (average) expected squared gradient norm to characterize the convergence rate.”