Variance-based Gradient Compression for Efficient Distributed Deep Learning

By Yusuke Tsuzuku, Hiroto Imachi, Takuya Akiba. 2018

91 citations recorded by the source corpus.

Due to the substantial computational cost, training state-of-the-art deep neural networks for large-scale datasets often requires distributed training using multiple computation workers. However, by nature, workers need to frequently communicate gradients, causing severe bottlenecks, especially on lower bandwidth connections. A few methods have been proposed to compress gradient for efficient communication, but they either suffer a low compression ratio or significantly harm the resulting model accuracy, particularly when applied to convolutional neural networks. To address these issues, we propose a method to reduce the communication overhead of distributed deep learning. Our key observation is that gradient updates can be delayed until an unambiguous (high amplitude, low variance) gradient has been calculated. We also present an efficient algorithm to compute the variance with negligible additional cost. We experimentally show that our method can achieve very high compression ratio while maintaining the result model accuracy. We also analyze the efficiency using computation and communication cost models and provide the evidence that this method enables distributed deep learning for many scenarios with commodity environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, background): “Strom [24] presents an approach (later modified by [26]) in which only gradients with a magnitude greater than a certain predefined threshold are sent to the server.”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “Motivated by the need for communication efficient protocols, various quantization algorithms have been proposed to reduce the communication cost [33, 20, 21, 38, 37, 34, 5].”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, background): “Other approaches for low-communication training focus on sparsification of gradients, either by thresholding small entries or by random sampling [6, 22, 23, 24, 25, 26, 27, 28].”
Deep Leakage from Gradients (2019, background): “Previous work [24, 36] show that gradients can be compressed by more than 300× without losing accuracy by error compensation techniques.”
Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air (2019, background): “Sparsification reduces the communication time by transmitting only some values of the gradient vectors [3], [25]– [31].”
Think Locally, Act Globally: Federated Learning with Local and Global Representations (2020, background): “Recent work has improved the communication efﬁciency of distributed learning by sparsifying the data and model [76], developing efﬁcient gradient-based methods [77, 21], and compressing the updates [75, 83, 49, 52, 46].”