Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted Aggregation

By Yang Chen, Xiaoyan Sun, Yaochu Jin. 2019

532 citations recorded by the source corpus.

Federated learning obtains a central model on the server by aggregating models trained locally on clients. As a result, federated learning does not require clients to upload their data to the server, thereby preserving the data privacy of the clients. One challenge in federated learning is to reduce the client–server communication since the end devices typically have very limited communication bandwidth. This article presents an enhanced federated learning technique by proposing an asynchronous learning strategy on the clients and a temporally weighted aggregation of the local models on the server. In the asynchronous learning strategy, different layers of the deep neural networks (DNNs) are categorized into shallow and deep layers, and the parameters of the deep layers are updated less frequently than those of the shallow layers. Furthermore, a temporally weighted aggregation strategy is introduced on the server to make use of the previously trained local models, thereby enhancing the accuracy and convergence of the central model. The proposed algorithm is empirically on two data sets with different DNNs. Our results demonstrate that the proposed asynchronous federated deep learning outperforms the baseline algorithm both in terms of communication cost and model accuracy.

Subfields: Artificial Intelligence, Building and Construction, Optimization & Training, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hangyu Zhu
2. Yaochu Jin
3. Han Yu
4. Shiqing Liu
5. Jinjin Xu
6. Qiang Yang
7. Zhen Fang
8. Azzam Mourad

In Building and Construction

1. Mohsen Guizani
2. Qiang Yang

In Optimization & Training

1. Yaochu Jin
2. Hangyu Zhu
3. Zhen Fang
4. Xiao Wang
5. Azzam Mourad
6. Dapeng Oliver Wu
7. Mohsen Guizani
8. Xiyao Ma

In Federated & Distributed Learning

1. Qiang Yang
2. Han Yu
3. Azzam Mourad
4. Li-zhen Cui
5. Hangyu Zhu
6. Xiao Wang
7. Mohsen Guizani
8. Alysa Ziying Tan

Citations in the corpus

• Distilled One-Shot Federated Learning (2020, methodology): “Asynchronous model updates have also been explored, using adaptive weighted averaging to combat staleness, combined with a proximal loss, [37] and updating deeper layer less frequently than shallower layers [4].”
• Federated Class-Incremental Learning (2022, methodology): “Moreover, [4] designs a layer-wise aggregation strategy to reduce computation overhead [55, 56].”
• A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, unknown): “Another strategy is to perform asynchronous training, i.e., updating the global model whenever it receives a model update from any of the participated clients [67], [165].”
• Towards Personalized Federated Learning (2021, background): “[97] and asynchronous model updates [98] are common strategies for addressing FL communication bottlenecks.”
• Federated Learning for Vehicular Internet of Things: Recent Advances and Open Issues (2020, methodology): “[12] propose a communication-efficient scheme for FL that employs a layerwise asynchronous update strategy”
• From distributed machine learning to federated learning: a survey (2021, methodology): “Multiple aggregation algorithms [30] are proposed to aggregate the models in distributed computing resources.”
• A Survey on Federated Learning: The Journey From Centralized to Distributed On-Site Learning and Beyond (2021, background): “The motivation of the work in [46] is to propose an enhanced FL framework, by not only reducing the communication cost but also improving the model accuracy.”
• Federated Learning on Non-IID Data: A Survey (2021, background): “A large body of research to address the above challenges, including reducing communication cost [18, 95, 97, 142], FL considering hardware constraints [28, 85], and additional protections against adversarial attacks [61, 160, 170].”