Federated Learning with Buffered Asynchronous Aggregation
By John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Michael G. Rabbat, Mani Malek, Dzmitry Huba. International Conference on Artificial Intelligence and Statistics, 2021
469 citations recorded by the source corpus.
Scalability and privacy are two critical concerns for cross-device federated learning (FL) systems. In this work, we identify that synchronous FL - synchronized aggregation of client updates in FL - cannot scale efficiently beyond a few hundred clients training in parallel. It leads to diminishing returns in model performance and training speed, analogous to large-batch training. On the other hand, asynchronous aggregation of client updates in FL (i.e., asynchronous FL) alleviates the scalability issue. However, aggregating individual client updates is incompatible with Secure Aggregation, which could result in an undesirable level of privacy for the system. To address these concerns, we propose a novel buffered asynchronous aggregation method, FedBuff, that is agnostic to the choice of optimizer, and combines the best properties of synchronous and asynchronous FL. We empirically demonstrate that FedBuff is 3.3x more efficient than synchronous FL and up to 2.5x more efficient than asynchronous FL, while being compatible with privacy-preserving technologies such as Secure Aggregation and differential privacy. We provide theoretical convergence guarantees in a smooth non-convex setting. Finally, we show that under differentially private training, FedBuff can outperform FedAvgM at low privacy settings and achieve the same utility for higher privacy settings.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/38fca90fef15b226dbae8db408c5efe8066dde1c
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yong Xiang
- Michael G. Rabbat
- Hongyuan Zhan
- Mani Malek
- John Nguyen
- Chenhao Xu
- Longxiang Gao
- Carole-Jean Wu
Citations in the corpus
- Asynchronous Federated Learning on Heterogeneous Devices: A Survey (2021, background): “In order to reduce the communication cost, more and more researchers work on deploying machine learning tasks on heterogeneous devices to enable smart interactions with humans [65].”
- Papaya: Practical, Private, and Scalable Federated Learning (2021, methodology): “In FedBuff, there is no notion of rounds: clients download, train, and upload updates asynchronously (Figure 4).”