Towards Federated Learning at Scale: System Design
By Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, A. Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konecný, Stefano Mazzocchi, H. Brendan McMahan and 4 more. 2019
3,255 citations recorded by the source corpus.
Federated Learning is a distributed machine learning approach which enables model training on a large corpus of decentralized data. We have built a scalable production system for Federated Learning in the domain of mobile devices, based on TensorFlow. In this paper, we describe the resulting high-level design, sketch some of the challenges and their solutions, and touch upon the open problems and future directions.
Subfields: Artificial Intelligence, Computer Science Applications, Efficiency & Compression, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Qiang Yang
- Dacheng Tao
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Ruslan Salakhutdinov
- Martin Jaggi
In Computer Science Applications
- Zachary B. Charles
- Peter Kairouz
- Hong-You Chen
- Wei-Lun Chao
- Ananda Theertha Suresh
- Gauri Joshi
- Jianyu Wang
- Ziteng Sun
In Efficiency & Compression
- Martin Jaggi
- Sebastian U. Stich
- Medhi Bennis
- Cong Xie
- Peter Richtárik
- Jakub Konecný
- Amir Salman Avestimehr
- Oluwasanmi Koyejo
In Optimization & Training
- Peter Richtárik
- Jakub Konecný
- Martin Jaggi
- Sebastian U. Stich
- Jianyu Wang
- Aryan Mokhtari
- Gauri Joshi
- Virginia Smith
Citations in the corpus
- Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “On-Device Learning On-device AI is becoming more ubiquitously adopted to enable model personalization [82], [83], [84] while improving data privacy [85], [86], [87], yet its impact in terms of carbon emission is often overlooked.”
- Advances and Open Problems in Federated Learning (2019, background): “[32] provides even more detail for a particular production system, including a discussion of specific architectural choices and considerations.”
- Federated Optimization in Heterogeneous Networks (2018, background): “vices to perform variable amounts of local work based on their underlying systems constraints; instead it is common to simply drop devices that fail to compute E epochs within a specified time window (Bonawitz et al., 2019). From a statistical perspective, FedAvg has been shown to diverge empirically”
- How To Backdoor Federated Learning (2018, background): “because federated learning is supposed to work with untrusted devices, a fraction of which may be malicious [6].”
- The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, background): “Federated Learning Under FL, a set of clients (e.g., mobile devices or institutions) collaboratively train a model under the orchestration of a central server, while keeping training data decentralized (McMahan et al., 2017; Bonawitz et al., 2019a).”
- Secure Single-Server Aggregation with (Poly)Logarithmic Overhead (2020, methodology): “That paper and subsequent work [7] showed that secure vector summation enables powerful privacy-preserving functionalities such as federated learning [25].”
- Flower: A Friendly Federated Learning Research Framework (2020, background): “Bonawitz et al. (2019) detail the system design of a large-scale Google-internal FL system.”
- IBM Federated Learning: an Enterprise Framework White Paper V0.1 (2020, background): “A few approaches and libraries for FL address some of these issues, including [9], [28], and [4].”
- Privacy Amplification via Random Check-Ins (2020, methodology): “However, existing approaches to combining FL and DP make a number of assumptions that are unrealistic in real-world FL deployments such as [11].”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “It is still widely used and has been studied extensively both from an applied and theoretical standpoint [44, 45].”
- Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy (2020, methodology): “Among such machine learning models we cite neural networks [31], linear [32, 33] and logistic [34] regression.”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “In many practical cross-device FL systems (see Bonawitz et al. (2019); Hard et al.”
- FedNAS: Federated Deep Learning via Neural Architecture Search (2020, methodology): “Our AutoFL system design is relevant to [1].”
- Adaptive Federated Optimization (2020, background): “In real cross-device FL deployments, communication bandwidth is a fundamental constraint (Bonawitz et al., 2019; Kairouz et al., 2019).”
- Salvaging Federated Learning by Local Adaptation (2020, background): “Designed for unbalanced, non-iid data distributions, federated learning has demonstrated good performance and scalability (Bonawitz et al., 2019) and is promoted by Google (Pichai, 2019) and other companies as the solution to privacy problems in predictive keyboards (Hard et al., 2018), medicine…”