FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
By Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, Ramtin Pedarsani. 2019
974 citations recorded by the source corpus.
Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systems-oriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance yet it remains challenging. In this paper, we present FedPAQ, a communication-efficient Federated Learning method with Periodic Averaging and Quantization. FedPAQ relies on three key features: (1) periodic averaging where models are updated locally at devices and only periodically averaged at the server; (2) partial device participation where only a fraction of devices participate in each round of the training; and (3) quantized message-passing where the edge nodes quantize their updates before uploading to the parameter server. These features address the communications and scalability challenges in federated learning. We also show that FedPAQ achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by our method.
Subfields: Artificial Intelligence, Efficiency & Compression, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Peter Richtárik
- Martin Jaggi
- Jakub Konecný
- Qiang Yang
- Sebastian U. Stich
- Ananda Theertha Suresh
- Peter Kairouz
- Yang Liu
In Efficiency & Compression
- Peter Richtárik
- Medhi Bennis
- Jakub Konecný
- Ahmed Khaled
- Oluwasanmi Koyejo
- Walid Saad
- H. Vincent Poor
- Dusist Niyato
In Federated & Distributed Learning
- Peter Richtárik
- Qiang Yang
- Jianyu Wang
- Jakub Konecný
- Gauri Joshi
- Martin Jaggi
- Zachary B. Charles
- Sebastian U. Stich
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, background): “The convergence analysis of local-update SGD methods is an active current area of research [370, 271, 428, 399, 334, 318, 233].”
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “Moreover, FedNova is also compatible with and complementary to gradient compression/quantization [43–48] and fair aggregation techniques [49, 50].”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “Finally, we note that a similar bound on the learning rate was used by Reisizadeh et al. (2020) in conjunction with a bounded variance assumption.”
- Personalized Federated Learning with Moreau Envelopes (2020, methodology): “To address the limitations on communications in a FL network, [10, 11] introduced quantization methods, while [12–14] proposed performing multiple local optimization rounds before sending the local models to the server.”
- Personalized Federated Learning: A Meta-Learning Approach (2020, methodology): “[10] A.”
- Distilled One-Shot Federated Learning (2020, background): “FedPAQ quantizes local updates before transmission, with averaging happens at both client and server sides [27].”
- Tighter Theory for Local SGD on Identical and Heterogeneous Data (2019, methodology): “…local stochastic gradient methods have been considered, for example with momentum (Yu et al., 2019a; Wang et al., 2019), with quantization (Reisizadeh et al., 2019; Basu et al., 2019), with adaptive stepsizes (Xie et al., 2019) and with various variance-reduction methods (Liang et al.,…”
- FedALA: Adaptive Local Aggregation for Personalized Federated Learning (2022, background): “This kind of data makes it hard to obtain a global model that generalizes to each client (McMahan et al. 2017; Reisizadeh et al. 2020; T Dinh, Tran, and Nguyen 2020).”
- Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, methodology): “For instance, FedPAQ (Reisizadeh et al., 2020), FedCOM (Had-dadpour et al., 2021), QSPARSE-SGD (Basu et al., 2019), Local-SGD-C (Gao et al., 2021) consider compressed FL algorithms under homogeneous data ( i.e. , iid clients).”
- Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, neutral): “quantization for FL were considered in [57]–[60].”
- Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challenges (2020, methodology): “Several quantization techniques (i.e., universal vector quantization [177], low-precision quantizer [178], and hyper-sphere quantization [179]) can be used for federated learning over IoT networks.”
- Server Averaging for Federated Learning (2021, background): “FedPAQ has made an initial effort [10] to periodically average and quantize the client models before making the server update.”
- Federated Learning for Internet of Things: A Comprehensive Survey (2021, methodology): “Then, all the clients transmit the trained parameters to the central server which aggregates them by using a weighted averaging algorithm such as Federated Averaging (FedAvg) [34].”
- Personalized Federated Learning using Hypernetworks (2021, background): “Duchi et al. (2014); McMahan et al. (2017b); Agarwal et al. (2018); Zhu et al. (2019) proposed new meth-ods for preserving privacy, and Reisizadeh et al. (2020); Dai et al. (2019); Basu et al. (2020); Li et al. (2020b); Stich (2018) focused on reducing communication cost.”
- Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2019, background): “FedPAQ [204] simultaneously incorporates these features and provides near-optimal theoretical guarantees for both strongly convex and non-convex loss functions, while empirically demonstrating the communication-computation tradeoff.”