Throughput-Optimal Topology Design for Cross-Silo Federated Learning

By Othmane Marfoq, Chuan Xu, Giovanni Neglia, Richard Vidal. 2020

113 citations recorded by the source corpus.

Federated learning usually employs a client-server architecture where an orchestrator iteratively aggregates model updates from remote clients and pushes them back a refined model. This approach may be inefficient in cross-silo settings, as close-by data silos with high-speed access links may exchange information faster than with the orchestrator, and the orchestrator may become a communication bottleneck. In this paper we define the problem of topology design for cross-silo federated learning using the theory of max-plus linear systems to compute the system throughput---number of communication rounds per time unit. We also propose practical algorithms that, under the knowledge of measurable network characteristics, find a topology with the largest throughput or with provable throughput guarantees. In realistic Internet networks with 10 Gbps access links for silos, our algorithms speed up training by a factor 9 and 1.5 in comparison to the master-slave architecture and to state-of-the-art MATCHA, respectively. Speedups are even larger with slower access links.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qinbin Li
2. Bingsheng He
3. Lichao Sun
4. Zhaomin Wu
5. Zeyi Wen
6. Philip S. Yu
7. Edwige Cyffers
8. Giovanni Neglia

In Electrical and Electronic Engineering

1. Bingsheng He
2. Philip S. Yu
3. Aurélien Bellet
4. Giovanni Neglia
5. Othmane Marfoq
6. Edwige Cyffers
7. Richard Vidal
8. Zeyi Wen

Citations in the corpus

• Privacy Amplification by Decentralization (2020, background): “cal analysis of (non-private) decentralized algorithms [37, 51, 35, 45, 40], owing to its simplicity and good empirical performance.”
• Federated Multi-Task Learning under a Mixture of Distributions (2021, background): “For these reasons, they have attracted significant interest recently in the machine learning community [41, 63, 42, 62, 3, 51, 46, 31].”
• Decentralized Federated Learning: A Survey and Perspective (2023, background): “DFL networks are inspired by various network topologies, as detailed in sources, such as [150], [151], [152], [153], and [154].”
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection (2019, methodology): “[118] propose an algorithm to find a throughput-optimal topology design.”