Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning
By XINYAN DAI, Xiao Yan, Kaiwen Zhou, Han Yang, Kelvin Kai Wing Ng, James Cheng, Yuwei Fan. 2019
51 citations recorded by the source corpus.
The high cost of communicating gradients is a major bottleneck for federated learning, as the bandwidth of the participating user devices is limited. Existing gradient compression algorithms are mainly designed for data centers with high-speed network and achieve $O(\sqrt{d} \log d)$ per-iteration communication cost at best, where $d$ is the size of the model. We propose hyper-sphere quantization (HSQ), a general framework that can be configured to achieve a continuum of trade-offs between communication efficiency and gradient accuracy. In particular, at the high compression ratio end, HSQ provides a low per-iteration communication cost of $O(\log d)$, which is favorable for federated learning. We prove the convergence of HSQ theoretically and show by experiments that HSQ significantly reduces the communication cost of model training without hurting convergence accuracy.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- H. Vincent Poor
- Mingzhe Chen
- Medhi Bennis
- Walid Saad
- Aryan Mokhtari
- Deniz Gündüz
- Gal Chechik
- Alireza Fallah
Citations in the corpus
- Personalized Federated Learning with Moreau Envelopes (2020, methodology): “To address the limitations on communications in a FL network, [10, 11] introduced quantization methods, while [12–14] proposed performing multiple local optimization rounds before sending the local models to the server.”
- Personalized Federated Learning: A Meta-Learning Approach (2020, methodology): “This includes quantization and compression ideas (Reisizadeh et al., 2019; Dai et al., 2019) as well as performing multiple local updates before communicating with the master (Stich, 2018; Lin et al., 2018; Wang and Joshi, 2018).”
- Personalized Federated Learning using Hypernetworks (2021, background): “Federated learning (FL) is the task of learning a model over multiple disjoint local datasets (McMahan et al., 2017a; Yang et al., 2019).”
- Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, background): “In [53], the authors designed a hyper-sphere quantization based FL algorithm so as to achieve a continuum of trade-offs between communication efficiency and gradient accuracy.”