Communication-efficient federated learning

By Mingzhe Chen, Nir Shlezinger, H. Vincent Poor, Yonina C. Eldar, Shuguang Cui. Proceedings of the National Academy of Sciences of the United States of America, 2021

358 citations recorded by the source corpus.

Significance Federated learning (FL) is an emerging paradigm that enables multiple devices to collaborate in training machine learning (ML) models without having to share their possibly private data. FL requires a multitude of devices to frequently exchange their learned model updates, thus introducing significant communication overhead, which imposes a major challenge in FL over realistic networks that are limited in computational and communication resources. In this article, we propose a communication-efficient FL framework that enables edge devices to efficiently train and transmit model parameters, thus significantly improving FL performance and convergence speed. Our proposed FL framework paves the way to collaborative ML in large-scale networking systems such as Internet of Things networks. Federated learning (FL) enables edge devices, such as Internet of Things devices (e.g., sensors), servers, and institutions (e.g., hospitals), to collaboratively train a machine learning (ML) model without sharing their private data. FL requires devices to exchange their ML parameters iteratively, and thus the time it requires to jointly learn a reliable model depends not only on the number of training steps but also on the ML parameter transmission time per step. In practice, FL parameter transmissions are often carried out by a multitude of participating devices over resource-limited communication networks, for example, wireless networks with limited bandwidth and power. Therefore, the repeated FL parameter transmission from edge devices induces a notable delay, which can be larger than the ML model training time by orders of magnitude. Hence, communication delay constitutes a major bottleneck in FL. Here, a communication-efficient FL framework is proposed to jointly improve the FL convergence time and the training loss. In this framework, a probabilistic device selection scheme is designed such that the devices that can significantly improve the convergence speed and training loss have higher probabilities of being selected for ML model transmission. To further reduce the FL convergence time, a quantization method is proposed to reduce the volume of the model parameters exchanged among devices, and an efficient wireless resource allocation scheme is developed. Simulation results show that the proposed FL framework can improve the identification accuracy and convergence time by up to 3.6% and 87% compared to standard FL.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/a3893072ac8aaaa01cbd754a59fabc4d6c1adb2c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Edge Learning for B5G Networks With Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing (2022, neutral): “can be optimized to improve the dual-functional performance metrics to facilitate learning [8], [9], [17].”
Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “In practical IoT applications, the devices are typically deployed in different network environments and have different network connectivity settings (3G, 4G, 5G, Wi-Fi) [26, 125], which leads to inconsistent communication bandwidth, latency, and reliability, i.e., communication heterogeneity [24].”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, neutral): “Their insight is that there are two stimulations for caching knowledge for FL [174]: i) training data sufficiency and ii) connectivity stability.”
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, background): “Representative Result: Next, we show a representative work in [48] that investigates the use of universal vector quantization for FedAvg.”
Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, background): “A probabilistic device scheduling policy was further proposed in [90], [243] to minimize the overall training time in wireless FL.”
Federated Learning for 6G: Applications, Challenges, and Opportunities (2021, background): “Therefore, the limited communication bandwidth may cause a longer communication delay, which can result in long convergence times for FL. Communication-efficient FL is thus an important area of current and future study [94–96].”
Federated Learning in Edge Computing: A Systematic Survey (2022, background): “Similarly, the authors of [169] proposed an FL framework to improve the FL convergence time and training”