Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

By Yujun Lin, Song Han, Huizi Mao, Yu Wang, William (Bill) J. Dally. 2017

1,676 citations recorded by the source corpus.

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Prior work also explores other forms of data parallelism [20, 23, 28], and there are opportunities to reduce data communication overheads of parallelism [5, 21, 37].”
Advances and Open Problems in Federated Learning (2019, background): “Consequently, a wider set of ideas related to error-correction such as [272, 346, 396, 380, 228, 371] are relevant in this setting, many of which could address both (a) and (b).”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “To tackle the system constraints challenge, researchers apply compression, sparsiﬁcation, or quantization techniques to reduce the communication overheads and computation costs during the training process [6, 7, 8, 9, 10, 11, 12].”
Backdoor attacks and defenses in feature-partitioned collaborative learning (2020, methodology): “We use the gradient sparsification algorithm proposed in (Lin et al., 2017), which adopts a drop rate at each iteration during training to threshold the gradients updated.”
rTop-k: A Statistical Estimation Approach to Distributed SGD (2020, methodology): “there is no compression, the setting proposed in [1], which uses the gradient top-k selection method, and the random-k sparsification strategy where the gradients to be communi-”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Gradient vector sparsification is another approach analyzed in [4, 40, 39, 5, 30, 34, 9, 3, 36, 21, 32].”
Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “The more direct approach is to reduce the size of the messages through gradient compression or sparsiﬁcation [32, 23, 51, 40, 42, 8].”
The Non-IID Data Quagmire of Decentralized Machine Learning (2019, methodology): “…these works is to reduce communication overhead among data partitions, and they either (i) assume the data partitions are IID (Hsieh et al., 2017; Shokri & Shmatikov, 2015; Lin et al., 2018) or (ii) conduct only a limited study on non-IID data partitions (McMahan et al., 2017; Tang et al., 2018).”
Decentralized Deep Learning with Arbitrary Communication Compression (2019, methodology): “To reduce the amount of data that has to be sent over each communication link, gradient compression techniques have been proposed for the standard distributed training case (Alistarh et al., 2017; Wen et al., 2017; Lin et al., 2018b; Wangni et al., 2018; Stich et al., 2018).”
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations (2019, methodology): “, where nodes locally select Topk values of the gradient in absolute value and transmit these at full precision [5], [14], [16]–[19], while maintaining errors in local nodes for later compensation.”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, methodology): “…(2014); Carlson et al. (2015); Bernstein et al. (2018, 2019) and Karimireddy et al. (2019) replace each coordinate of the gradient with its sign; Lin et al. (2018); Stich et al. (2018) and Wangni et al. (2018) use the largest few coordinates; and Koneˇcn ` y et al. (2016) and Wang et al. (2018)…”
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, methodology): “[25] present minor modifications to the work of Aji and Heafield [23] that even close this small”
Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication (2019, methodology): “Here we propose the ﬁrst method that supports arbitrary low accuracy and even biased compression operators, such as in (Alistarh et al., 2018; Lin et al., 2018; Stich et al., 2018).”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, methodology): “Though not presented as such, the ‘momentum correction’ used in Lin et al. [2018] is a variant of error-feedback.”
Federated Machine Learning (2019, background): “ods to improve the communication cost to facilitate the training of centralized models based on data distributed over mobile clients. Recently, a compression approach called Deep Gradient Compression [39] is proposed to greatly reduce the communication bandwidth in large-scale distributed training. ”
Split learning for health: Distributed deep learning without sharing raw patient data (2018, background): “This uneven distri-3 [36, 37, 38] for seamless distributed learning with edge devices.”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “The resulting communication bottleneck (Keuper & Preundt, 2016; Lin et al., 2017) however severely limits scalability with the number of workers (e.”
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms (2018, background): “An alternative approach to communication-efficiency is gradient compression techniques (Wen et al., 2017; Lin et al., 2017) that quantize the gradients computed by workers.”