PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization

By Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi. 2019

446 citations recorded by the source corpus.

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on power iteration that can i) compress gradients rapidly, ii) efficiently aggregate the compressed gradients using all-reduce, and iii) achieve test performance on par with SGD. The proposed algorithm is the only method evaluated that achieves consistent wall-clock speedups when benchmarked against regular SGD with an optimized communication backend. We demonstrate reduced training times for convolutional networks as well as LSTMs on common datasets. Our code is available at this https URL.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Optimization & Training

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “In addition, significant research has gone into algorithmic approaches to efficiently scale training [61], [62] by reducing communication cost via compression [63], [64], pipelining [65], and sharding [66], [67].”
Zero-Shot Text-to-Image Generation (2021, methodology): “We use PowerSGD (Vogels et al., 2019) to compress the gradients with respect to all parameters except the embeddings, unembeddings, gains, and biases.”
Advances and Open Problems in Federated Learning (2019, background): “Consequently, a wider set of ideas related to error-correction such as [272, 346, 396, 380, 228, 371] are relevant in this setting, many of which could address both (a) and (b).”
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum (2019, background): “Note that various approaches to gradient compression have been proposed to reduce the communication overhead for ALLREDUCE and decentralized learning methods (Alistarh et al., 2007; Wen et al., 2007; Bernstein et al., 2019; Karimireddy et al., 2019; Koloskova et al., 2019b; Vogels et al., 2019).”
Natural Compression for Distributed Deep Learning (2019, methodology): “We evaluate TopK-Cnat following Table 4 in (Vogels et al., 2019).”
This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. (unknown): “In addition to parallelization, there are other complementary techniques for training large-scale models, such as memory optimization [12, 14,21,23,28,46], communication compression [6,53], and low-precision training [35].”
Distributed Learning with Sparse Communications by Identification (2018, background): “to early works [55, 61] and more recent developments [23, 59, 1, 4, 66] among many others.”
CSER: Communication-efficient SGD with Error Reset (2020, methodology): “In this paper, we use a blockwise random sparsifier with synchronized random seed, which is also mentioned in [23].”
Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM (2020, background): “…communication payload size per link, model updates are quantized under centralized (Bernstein et al., 2018; Suresh et al., 2017; Sun et al., 2019; Vogels et al., 2019; Alistarh et al., 2017; Horv´ath et al., 2019) and decentralized network topologies (Sriranga et al., 2019; Zhu et al., 2016;…”
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed (2021, unknown): “…the promising efﬁciency of this method, error compensation has been applied into many related area (Zheng et al., 2019; Phuong and Phong, 2020; Yu et al., 2019; Shi et al., 2019; Ivkin et al., 2019; Sun et al., 2019; Basu et al., 2019; Vogels et al., 2019) in order to reduce the communication cost.”