Expanding the Reach of Federated Learning by Reducing Client Resource Requirements

By Sebastian Caldas, Jakub Konecný, H. Brendan McMahan, Ameet Talwalkar. 2018

524 citations recorded by the source corpus.

Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent server-to-client; and (2) Federated Dropout, which allows users to efficiently train locally on smaller subsets of the global model and also provides a reduction in both client-to-server communication and local computation. We empirically show that these strategies, combined with existing compression approaches for client-to-server communication, collectively provide up to a $14\times$ reduction in server-to-client communication, a $1.7\times$ reduction in local computation, and a $28\times$ reduction in upload communication, all without degrading the quality of the final model. We thus comprehensively reduce FL's impact on client device resources, allowing higher capacity models to be trained, and a more diverse set of users to be reached.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Efficiency & Compression, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Efficiency & Compression

In Federated & Distributed Learning

Citations in the corpus

Breaking the Communication-Privacy-Accuracy Trilemma (2020, background): “While various compression schemes, based on quantization, sparsification, and dithering have been proposed in the recent literature and Kashin’s representation for communication efficiency [27], [28], [29], [30] or for LDP [31] has also been explored in a few works, it is particularly powerful in th”
Distributed Fixed Point Methods with Compressed Iterates (2019, methodology): “[9, 20], where concerns of communication efficiency and memory usage are particularly important.”
Secure Federated Submodel Learning (2019, background): “Another effective approach to improving communication efficiency is to first apply dropout strategies to the global model, and then let clients train over the same reduced model architecture [31].”
Federated Learning with Autotuned Communication-Efficient Secure Aggregation (2019, methodology): “We note that logarithmic dependence on d in the above MSE bound can be replaced with a constant [4], by appropriate use of Kashin’s representation [5].”
Improving Federated Learning Personalization via Model Agnostic Meta Learning (2019, methodology): “We suggest these objectives become the subject of MAML works, in addition to the performance after adaptation, and to consider the datasets with a natural user/client structure being established for Federated Learning (Caldas et al., 2018b) as the source of experiments for supervised learning.”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, neutral): “With a parameter server, both the ‘clients → server’ and ‘server → clients’ communication have to be compressed (Caldas et al., 2018; Bernstein et al., 2019; Seide et al., 2014).”
Towards Federated Learning at Scale: System Design (2019, methodology): “To reduce the bandwidth necessary, we implement compression techniques such as those of Kone ˇ cn ´ y et al. (2016b) and Caldas et al. (2018).”
Federated Learning: Challenges, Methods, and Future Directions (2019, neutral): “54 IEEE SIGNAL PROCESSING MAGAZINE | May 2020 | structured random rotations [22], and using lossy compression and dropout to reduce server-to-device communication [9].”
FedBoost: Communication-Efficient Algorithms for Federated Learning (2020, unknown): “Communication-efficient algorithms for distributed optimization has been the focus of several studies, both in federated learning (Konečný et al., 2016b;a; Suresh et al., 2017; Caldas et al., 2018) and in other distributed settings (Stich et al.”
TiFL: A Tier-based Federated Learning System (2020, methodology): “We use the standard model architecture as provided in LEAF [6].”
A Survey on Federated Learning: The Journey From Centralized to Distributed On-Site Learning and Beyond (2021, methodology): “The communication cost minimization, presented in [43], aims to reduce the size of the models generated by both the server and the clients, while updating the FL process as follows.”
Model Pruning Enables Efficient Federated Learning on Edge Devices (2019, methodology): “To reduce both communication and computation costs, efﬁcient FL techniques using lossy compression and dropout were developed [8], [10], where the ﬁnal model still has the original size, hence providing no beneﬁt for efﬁcient inference after the model is trained.”
Federated Learning in Mobile Edge Networks: A Comprehensive Survey (2019, methodology): “The authors in [94] extend the studies in [89] by proposing lossy compression and federated dropout to reduce server-toparticipant communication costs.”
A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “With the same motivation, Caldas et al. [150] applied lossy compression and federated dropout to train a smaller sub-set of local clients and reduce client-to-server interaction and local computation (see Fig.”
Better Communication Complexity for Local SGD (2019, background): “Due to the need to solve such problems more efﬁciently, there has been a lot of recent interest in understanding the trade-offs between communication and computation, a concern which is particularly important in the federated learning setting; see [3, 10, 13].”