GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training

By Mingchao Yu, Zhifeng Lin, Krishnagiri Narra, Songze Li, Youjie Li, Nam Sung Kim, Alexander Gerhard Schwing, Murali Annavaram, Amir Salman Avestimehr. 2018

70 citations recorded by the source corpus.

Data parallelism can boost the training speed of convolutional neural networks (CNN), but could suffer from significant communication costs caused by gradient aggregation. To alleviate this problem, several scalar quantization techniques have been developed to compress the gradients. But these techniques could perform poorly when used together with decentralized aggregation protocols like ring all-reduce (RAR), mainly due to their inability to directly aggregate compressed gradients. In this paper, we empirically demonstrate the strong linear correlations between CNN gradients, and propose a gradient vector quantization technique, named GradiVeQ, to exploit these correlations through principal component analysis (PCA) for substantial gradient dimension reduction. GradiVeQ enables direct aggregation of compressed gradients, hence allows us to build a distributed learning system that parallelizes GradiVeQ gradient compression and RAR communications. Extensive experiments on popular CNNs demonstrate that applying GradiVeQ slashes the wall-clock gradient aggregation time of the original RAR by more than 5X without noticeable accuracy loss, and reduces the end-to-end training time by almost 50%. The results also show that GradiVeQ is compatible with scalar quantization techniques such as QSGD (Quantized SGD), and achieves a much higher speed-up gain under the same compression ratio.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, background): “Commutative compression and addition Yu et al. (2018) stress that commutability of compression with gradient addition enables efﬁcient aggregation with ring all-reduce .”
Communication-Efficient Edge AI: Algorithms and Systems (2020, methodology): “As pointed out in [175], [176], scalar quantization methods fail under decentralized networks without a central aggregation node.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “In the scenario of distributed training, Yu et al. [91] utilized principal component analysis (PCA) to linearly project the weight gradients into a low-dimensional space that enables fast decentralized gradient aggregation (e.g., ring all-reduce) in the compressed domain. c) Other matrix…”
This paper is included in the Proceedings of the 19th USENIX Symposium on Networked Systems Design and Implementation. (methodology): “…on mobile devices [37, 40], per-layer heterogeneous quantization of DNNs [39], mixed precision quantization that adapts to underlying hardware capabilities [34], quantization of gradient vectors for bandwidth efﬁcient aggregation [1,9,36], lossy training using 1-bit quantization [29] and more.”
OD-SGD (2020, methodology): “Mingchao Yu et al. [41] propose a gradient vector quantization technique with the name GradiVeQ by using the strong linear correlations between CNN gradients.”
Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning (2019, methodology): “GradiVeQ (Yu et al., 2018) compresses the gradient g using its projections on the eigenvectors (that corresponds to large eigenvalues) of the gradient covariance matrix.”