TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

By Wujie Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Helen Li. 2017

1,084 citations recorded by the source corpus.

High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {-1,0,1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet does not incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. Our source code is available.

Subfields: Artificial Intelligence, Control and Systems Engineering, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Electrical and Electronic Engineering

In Optimization & Training

Citations in the corpus

Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Prior work also explores other forms of data parallelism [20, 23, 28], and there are opportunities to reduce data communication overheads of parallelism [5, 21, 37].”
Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs (2020, background): “There has been a signiﬁcant recent progress in this topic (see [AGL + 17,KRSJ19,WXY + 17,SCJ18,AHJ + 18,KLSJ19,SCJ18,BDKD19,SDGD19,SDGD20] and references therein).”
Distributed Fixed Point Methods with Compressed Iterates (2019, methodology): “To overcome this issue, standard techniques proceed by compressing the gradients of some SGD (Stochastic Gradient Descent) like training algorithm as in [1, 7, 32].”
Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates (2019, methodology): “Various approaches have been proposed to reduce the communication overhead of distributed SGD, such as quantization (Seide et al., 2014; Strom, 2015; Wen et al., 2017; Alistarh et al., 2016; Bernstein et al., 2018; Karimireddy et al., 2019; Zheng et al., 2019) and sparsification (Aji and Heafield,…”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “In [4, 40, 44], random quantization schemes are studied.”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, background): “Note that various approaches to gradient compression have been proposed to reduce the communication overhead for ALLREDUCE and decentralized learning methods (Alistarh et al., 2007; Wen et al., 2007; Bernstein et al., 2019; Karimireddy et al., 2019; Koloskova et al., 2019b; Vogels et al., 2019).”
Gradient Descent with Compressed Iterates (2019, methodology): “As a result, there are many algorithms designed with the goal of reducing communication in stochastic gradient methods: including SignSGD (1-bit quantization) [4], TernGrad (ternary quantization) [35], QSGD [1], DIANA (with arbitrary quantization) [11], ChocoSGD [15], and others, see e.”
Decentralized Deep Learning with Arbitrary Communication Compression (2019, methodology): “Gradient compression techniques have been proposed for the standard distributed training case (Alistarh et al., 2017; Wen et al., 2017; Lin et al., 2018; Wangni et al., 2018; Stich et al., 2018), to reduce the amount of data that has to be sent over each communication link in the network.”
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression (2019, background): “Previous works mostly focus on a centralized scenario where the compression is assumed to be unbiased [Jiang and Agrawal, 2018, Shen et al., 2018, Wangni et al., 2018, Wen et al., 2017, Zhang et al., 2017].”
QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “While previous work [2, 4, 9, 24, 30, 36, 46, 56] had already introduced the idea of functions Qw ,Qa to quantize weights and/or activations in the forward pass, they required the weights W or gradients G, e to be represented in floating-point in the backward pass, in order to optimize accurately.”
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations (2019, methodology): “Theoretically justified gradient compression using unbiased stochastic quantizers has been proposed and analyzed in [6], [8], [13], [15].”
Natural Compression for Distributed Deep Learning (2019, background): “, quantization, sparsification, dithering) techniques [2, 47, 48, 22] have been proposed in recent years as a successful tool to alleviate this problem.”
MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling (2019, background): “There have been massive amount of work in the context of algorithms [37, 33, 36, 5, 26] and systems [16, 43, 9] that improve the communication efficiency of synchronous distributed SGD in a fully-connected network.”
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression (2019, methodology): “ssed PCA (Garber et al., 2017), are also studied recently. In Seide et al. (2014), an 1Bit-SGD was proposed to utilize only the sign of each element in the gradient vector for stochastic gradient. In Wen et al. (2017), authors manipulate the 1Bit-SGD to ensure that the compressed is an unbiased esti”
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization (2019, methodology): “To reduce the communication cost at each iteration, compression or sparsification based parallel SGD are considered in (Seide et al., 2014; Strom, 2015; Aji & Heafield, 2017; Wen et al., 2017; Alistarh et al., 2017).”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, background): “, 2019), quantizing the individual coordinates (Alistarh et al., 2017; Wen et al., 2017), and low-rank approximation of the gradient (Wang et al.”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the work in this direction can be found in Zhang et al. (2017); Wen et al. (2017); Wangni et al. (2017); Sa et al. (2015); Na et al. (2017); Gupta et al. (2015); Alistarh et al. (2016); Khirirat et al. (2018).”
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, background): “the downlink typically is cheaper and has far higher bandwidth than the uplink, as already noted in [10] and [19].”