QSGD: Randomized Quantization for Communication-Optimal Stochastic Gradient Descent

By Dan Alistarh, Jerry Zheng Li, Ryota Tomioka, Milan Vojnovic. 2016

125 citations recorded by the source corpus.

Subfields: Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the work in this direction can be found in Zhang et al. (2017); Wen et al. (2017); Wangni et al. (2017); Sa et al. (2015); Na et al. (2017); Gupta et al. (2015); Alistarh et al. (2016); Khirirat et al. (2018).”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “, see [21] and in particular [5] showed that stochastic quantization and Elias coding [15] can be used to obtain communication-optimal SGD for convex functions.”
Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training (2017, methodology): “Alistarh et al. (2016) proposed another approach called QSGD which balance the trade-off between accuracy and gradient precision.”
Distributed Mean Estimation with Limited Communication (2016, neutral): “Recently, algorithms involving distributed mean estimation have been used extensively in training large-scale neural networks and other statistical models (McDonald et al., 2010; Povey et al., 2014; Dean et al., 2012; McMahan et al., 2016; Alistarh et al., 2016).”
Model compression via distillation and quantization (2018, methodology): “Alistarh et al. (2016), that is, we will apply the scaling function separately to buckets of consecutive values of a certain fixed size.”
Gradient Sparsification for Communication-Efficient Distributed Optimization (2017, methodology): “We compared the gradient sparsification method with the quantized sparse gradient descent (QSGD) algorithm in [1].”
Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform (2018, background): “The communication overhead severely hinders the scalability of data parallelism [18, 19, 20, 21, 22, 14].”
Gradient Diversity: a Key Ingredient for Scalable Distributed Learning (2017, methodology): “A few other algorithms have been recently proposed to reduce the communication cost by inducing sparsity in the gradients, for instance, QSGD (Alistarh et al., 2016) and Tern-Grad (Wen et al., 2017).”
ResizeMix: Mixing Data with Preserved Object Information and True Labels (2020, background): “Recent optimization techniques have further promoted the CNN performance to a new level, including data augmentation [8, 51], optimizer design [1, 30], learning rate sched-”
ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning (2017, background): “When l(·, ·) is the least squares loss, we first notice that simply doing stochastic quantization of data samples (i.e., Qs) introduces bias of the gradient estimator and therefore SGD would converge to a different solution.”
The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning (2016, unknown): “4 in Alistarh et al. (2016). The proofs follow immediately from those in Alistarh et al. (2016), and so we omit them for conciseness.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “[ Alistarh et al. , 2016 ] also considered the problem of gradient communication in the parallel SGD.”
SGD: Decentralized Byzantine Resilience (2019, background): “Indeed, using imprecise or mildly diverging parameters have proven themselves beneﬁcial in terms of convergence speed and quality [44, 6, 26].”
A Hitchhiker's Guide On Distributed Training of Deep Neural Networks (2018, methodology): “[47] proposed Quantized SGD (QSGD) which quantizes components to discrete set of values without losing statistical properties using stochastical rounding to generate lossless encoding of the output.”
Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutes (2018, background): “This is a part of a larger research aiming at gradient quantization i.e., reducing the precision of the communicated values [4] [27]).”