Federated Learning: Challenges, Methods, and Future Directions
By Tian Li, Anit Kumar Sahu, Ameet Talwalkar, Virginia Smith. 2019
6,159 citations recorded by the source corpus.
Federated learning involves training statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping data localized. Training in heterogeneous and potentially massive networks introduces novel challenges that require a fundamental departure from standard approaches for large-scale machine learning, distributed optimization, and privacy-preserving data analysis. In this article, we discuss the unique characteristics and challenges of federated learning, provide a broad overview of current approaches, and outline several directions of future work that are relevant to a wide range of research communities.
Subfields: Artificial Intelligence, Information Systems, Transportation, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Peter Richtárik
- Dacheng Tao
- Qiang Yang
- Martin Jaggi
- Jakub Konecný
- Percy Liang
- Yoshua Bengio
- Sergey Levine
In Information Systems
- Dusist Niyato
- Peter Richtárik
- Zachary B. Charles
- Mingxia Liu
- Jakub Konecný
- Dinh C. Nguyen
- Pubudu N. Pathirana
- Zehui Xiong
In Transportation
- Peter Richtárik
- Jakub Konecný
- Zachary B. Charles
- Aryan Mokhtari
- Dusist Niyato
- Filip Hanzely
- Dmitrii Avdiukhin
- Jianwei Ma
In Federated & Distributed Learning
- Peter Richtárik
- Aryan Mokhtari
- Mehrdad Mahdavi
- Virginia Smith
- Zachary B. Charles
- Jakub Konecný
- Bingsheng He
- Qinbin Li
Citations in the corpus
- Federated Optimization in Heterogeneous Networks (2018, background): “While there is a wealth of work on distributed optimization in the context of machine learning, two key challenges distinguish federated learning from traditional distributed optimization: high degrees of systems and statistical heterogeneity1 (McMahan et al., 2017; Li et al., 2019).”
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “Federated Learning (FL) has emerged as an important machine learning paradigm in which a federation of clients participate in collaborative training of a centralized model (Shokri & Shmatikov, 2015; McMahan et al., 2016; Smith et al., 2017; Caldas et al., 2018; Bonawitz et al., 2019; Li et al., 2019”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Related Work Due to its emphasis on preserving privacy, FL has gained significant attention in the community [12, 39, 40, 41, 42].”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “For example, while the FedProx algorithm (Li et al., 2019) does”
- Adaptive Federated Optimization (2020, background): “For a more in-depth discussion of federated learning and the challenges therein, we defer to Kairouz et al. (2019) and Li et al. (2019a).”
- Personalized Federated Learning: A Meta-Learning Approach (2020, background): “Recently we have witnessed significant progress in developing novel methods that address different challenges in FL; see [4, 5].”
- Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “Several other works have also demonstrated that leveraging the information that is distributed over many mobile users can increase the training performance dramatically, while ensuring data privacy and locality [9]–[11].”
- Robust Aggregation for Federated Learning (2019, background): “Extensions were proposed in [7, 29, 33, 47, 50, 57, 69, 75, 79]; see also the recent surveys [46, 56].”
- Secure Federated Submodel Learning (2019, background): “For more related work, we direct interested readers to the surveys written by Li et al. [22] and Yang et al. [23].”
- Communication Efficient Decentralized Training with Multiple Local Updates (2019, background): “Federated Learning (FL), which has a central parameter server, enables massive edge computing devices to jointly learn a centralized model while keeping all local data localized [13, 34, 26, 12, 18, 30, 57].”
- SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning (2019, methodology): “The FedProx algorithm by Li et al. (2019a) seeks to minimize the local drift by imposing additional regularization on each client.”
- FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, methodology): “Federated Learning is a novel paradigm that aims to train a statistical model at the “edge” nodes as opposed to the traditional distributed computing systems such as data centers [Konečn`y et al., 2016, Li et al., 2019a].”
- Improving Federated Learning Personalization via Model Agnostic Meta Learning (2019, background): “Theoretical guarantees are only available under restrictive assumptions and for convex objectives, cf. Li et al. (2019b).”
- Differentially Private Meta-Learning (2019, background): “Finally, we show an application of our work by drawing connections to federated learning (FL) (Li et al., 2019).”
- Gradient Descent with Compressed Iterates (2019, methodology): “This situation is common in large-scale distributed optimization involving many edge devices, and common challenges include data heterogeneity [37], privacy [7], resource management [25, 33], and system heterogeneity as well as communication efficiency [16, 18].”