Asynchronous Federated Optimization
By Cong Xie, Oluwasanmi Koyejo, Indranil Gupta. 2019
771 citations recorded by the source corpus.
Federated learning enables training on a massive number of edge devices. To improve flexibility and scalability, we propose a new asynchronous federated optimization algorithm. We prove that the proposed approach has near-linear convergence to a global optimum, for both strongly and non-strongly convex problems, as well as a restricted family of non-convex problems. Empirical results show that the proposed algorithm converges fast and tolerates staleness.
Subfields: Artificial Intelligence, Optimization & Training, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Optimization & Training
- Jianyu Wang
- Gauri Joshi
- Hao Liang
- H. Vincent Poor
- Michael G. Rabbat
- Shiqiang Wang
- Qinghua Liu
- Yong Xiang
In Federated & Distributed Learning
Citations in the corpus
- Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization (2020, methodology): “Besides, asynchronous federated optimization algorithms proposed in [41, 9] take a different approach of allowing clients make updates to stale versions of the global model, and their analyses are limited to IID local datasets and convex local functions.”
- Distilled One-Shot Federated Learning (2020, methodology): “Asynchronous model updates have also been explored, using adaptive weighted averaging to combat staleness, combined with a proximal loss, [35] and updating deeper layer less frequently than shallower layers [4].”
- Adaptive Task Allocation for Asynchronous Federated Mobile Edge Learning (2019, methodology): “ learner. Recently, some work has been carried out on allowing some staleness so that powerful devices with good communicationlinks may actually provide a faster validation accuracy progression [12], [13]. The works of [6], [7] aimed to optimize the number of local epochs per node with respect to to”
- SAFA: A Semi-Asynchronous Protocol for Fast Federated Learning With Low Overhead (2019, methodology): “ness [9]) on the global learning progress.”
- Asynchronous Online Federated Learning for Edge Devices with Non-IID Data (2019, methodology): “The proposed ASO-Fed inherits the idea of using asynchronous update scheme as [8], [9].”
- Asynchronous Federated Learning on Heterogeneous Devices: A Survey (2021, background): “In [100], a mixing hyperparameter is proposed to balance the convergence rate with variance reduction according to the staleness.”
- Federated Learning: Opportunities and Challenges (2021, background): “Works such as [29, 34, 35] talks about carrying out federated learning in an asynchronous environment.”
- A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, background): “The communication between the manager and the local parties can be synchronous [129] or asynchronous [204, 171].”
- Papaya: Practical, Private, and Scalable Federated Learning (2021, methodology): “AsyncFL methods have been previously explored (Xie et al., 2019; Nguyen et al., 2021; Xu et al., 2021), but none has yet been demonstrated and evaluated at scale.”
- Federated Learning with Buffered Asynchronous Aggregation (2021, methodology): “To control the effect of staleness τi(t) in client i’s contribution to the t-th server update, we adopt a polynomial staleness function, s(τi(t)) := 1/(1 + t), as suggested in (Xie et al., 2019) to adaptively discount stale client terms when they are aggregated at the server.”
- Convergence of Edge Computing and Deep Learning: A Comprehensive Survey (2019, background): “These bottlenecks in FL can potentially be addressed by asynchronous training mechanisms [240], [241].”
- A Survey on Federated Learning for Resource-Constrained IoT Devices (2021, positive): “Xie et al. [96] proposed asynchronous federated optimization, [159] discussed on the asynchronous FL for geospatial applications, [160]–[162] proposed a DP-based asynchronous FL strategy for MEC, and [163] presented a blockchain-based secure data sharing strategy for asynchronous FL.”
- Federated Learning in Mobile Edge Networks: A Comprehensive Survey (2019, methodology): “As such, asynchronous FL has been proposed as a solution in [114] and [134].”