Robust Aggregation for Federated Learning
By Krishna Pillutla, Sham M. Kakade, Zaïd Harchaoui. 2019
1,001 citations recorded by the source corpus.
We present a novel approach to federated learning that endows its aggregation process with greater robustness to potential poisoning of local data or model parameters of participating devices. The proposed approach, Robust Federated Aggregation (RFA), relies on the aggregation of updates using the geometric median, which can be computed efficiently using a Weiszfeld-type algorithm. RFA is agnostic to the level of corruption and aggregates model updates without revealing each device’s individual contribution. We establish the convergence of the robust federated learning algorithm for the stochastic learning of additive models with least squares. We also offer two variants of RFA: a faster one with one-step robust aggregation, and another one with on-device personalization. We present experimental results with additive models and deep networks for three tasks in computer vision and natural language processing. The experiments show that RFA is competitive with the classical aggregation when the level of corruption is low, while demonstrating greater robustness under high corruption.
Subfields: Artificial Intelligence, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Martin Jaggi
- Qiang Yang
- Peter Richtárik
- Jakub Konecný
- Percy Liang
- Yang Liu
- Sebastian U. Stich
- Quoc V. Le
In Federated & Distributed Learning
- Qiang Yang
- Jianyu Wang
- Gauri Joshi
- Virginia Smith
- Martin Jaggi
- Chaoyang He
- Murali Annavaram
- Jakub Konecný
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, methodology): “ [426]. Past work has shown that various robust aggregators are provably effective for Byzantine-tolerant distributed learning [372, 69, 101] under appropriate assumptions, even in federated settings [326, 415]. Despite this, Fang et al. [161] recently showed that multiple Byzantine-resilient defens”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “To accelerate generating benchmark results on new types of adversarial attacks in FL, we include the latest robust aggregation methods presented in literature i.e. (i) norm difference clipping [23]; weak differential private (DP) [23]; (ii) RFA (geometric median) [118]; (iii) K RUM and (iv) M ULTI -”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “Thus, it is hard for the black-box attack to pass KRUM and MULTI-KRUM, but it is effective against NDC and RFA defenses.”
- Adaptive Personalized Federated Learning (2020, methodology): “Motivated by this goal, most of the existing methods pursue the following procedure to learn a global model: (i) a subset of clients participating in the training is chosen at each round and receive the current copy of the global model; (ii) each chosen client updates the local version of the global”
- Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning (2021, methodology): “, by circumventing the security protocols of operating systems such as Android) and can directly manipulate the poisoned updates [5], [10], [23], [41], [50], [55].”
- An Overview of Federated Deep Learning Privacy Attacks and Defensive Strategies (2020, background): “[75] proposed an approximate geometric median for the use in federated learning and demonstrated its effectiveness by introducing model corrupting updates and data.”
- Robust Federated Recommendation System (2020, methodology): “4 n , and evaluate and compare our A-RFRS with RFA, Trmean, Krum and No defense model (our non-robust A-FRS) on four different datasets.”
- Federated learning and differential privacy for medical image analysis (2021, methodology): “Pillutla et al. 10 proposed a robust and secure aggregation oracle based on the geometric median using a constant number of calls to a regular non-robust secure average oracle.”
- Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges (2022, background): “• Geometric-mean [88, 89]: It represents the central tendency or the typical value of the data distribution by using the product of their values.”
- Byzantine-Robust Learning on Heterogeneous Datasets via Resampling (2020, methodology): “R FA (Pillutla et al., 2019)).”
- Ditto: Fair and Robust Federated Learning Through Personalization (2020, background): “In terms of defenses, robust aggregation is a common strategy to mitigate the effect of malicious updates (Blanchard et al., 2017; Pillutla et al., 2019; Sun et al., 2019; Li et al., 2019; He et al., 2020).”
- From distributed machine learning to federated learning: a survey (2021, background): “, differential privacy [121], homomorphic encryption [59], and robustness aggregation [142], are designed to address the possible attacks [49,71,181].”
- Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “Another class of approaches employs the coordinate-wise median gradient [192], geometric median of means [193], and approximate geometric median [194] since median-based computations are more resistant to outliers than mean-based aggregation.”
- Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies (2020, background): “An interesting future direction is to improve the fairness [34–37] and robustness [38] of POWER-OF-CHOICE by modifying step 3 of the algorithm to use a different metric such as the clipped loss or the q-fair loss proposed [34] instead of Fk(w).”