Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection

By Zhenheng Tang, Shaohuai Shi, Xiaowen Chu. 2020

72 citations recorded by the source corpus.

The increasing size of machine learning models, especially deep neural network models, can improve the model generalization capability. However, large models require more training data and more computing resources (such as GPU clusters) to train. In distributed training, the communication overhead of exchanging gradients or models among workers becomes a potential system bottleneck that limits the system scalability. Recently, many research works aim to reduce communication time of two types of distributed deep learning architectures, centralized and decentralized.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qiang Yang
2. Murali Annavaram
3. Chaoyang He
4. Yang Liu
5. Philip S. Yu
6. S. Avestimehr
7. Lichao Sun
8. Praneeth Vepakomma

In Electrical and Electronic Engineering

1. Philip S. Yu
2. Jinhyun So
3. Murali Annavaram
4. Ramesh Raskar
5. Yan Kang
6. Li Shen
7. Qiang Yang
8. Abhishek Singh

Citations in the corpus

• FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “To tackle the system constraints challenge, researchers apply compression, sparsification, or quantization techniques to reduce the communication overheads and computation costs during the training process [6, 7, 8, 9, 10, 11, 12].”
• Decentralized Federated Learning: A Survey and Perspective (2023, background): “Some researchers have also focused on addressing the bandwidth differences among different communication lines [219], [220].”