Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
By Hongyi Wang, Kartik K. Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, Dimitris Papailiopoulos. 2020
829 citations recorded by the source corpus.
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training. The goal of a backdoor is to corrupt the performance of the trained model on specific sub-tasks (e.g., by classifying green cars as frogs). A range of FL backdoor attacks have been introduced in the literature, but also methods to defend against them, and it is currently an open question whether FL systems can be tailored to be robust against backdoors. In this work, we provide evidence to the contrary. We first establish that, in the general case, robustness to backdoors implies model robustness to adversarial examples, a major open problem in itself. Furthermore, detecting the presence of a backdoor in a FL model is unlikely assuming first order oracles or polynomial time. We couple our theoretical results with a new family of backdoor attacks, which we refer to as edge-case backdoors. An edge-case backdoor forces a model to misclassify on seemingly easy inputs that are however unlikely to be part of the training, or test data, i.e., they live on the tail of the input distribution. We explain how these edge-case backdoors can lead to unsavory failures and may have serious repercussions on fairness, and exhibit that with careful tuning at the side of the adversary, one can insert them across a range of machine learning tasks (e.g., image classification, OCR, text prediction, sentiment analysis).
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Safety, Robustness & Interpretability
- Bo Li
- Dawn Song
- Xingjun Ma
- Tom Goldstein
- Aleksander Madry
- Nicholas Carlini
- Ajmal Saeed Mian
- Dimitris Tsipras
In Federated & Distributed Learning
Citations in the corpus
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “In standard FL settings, it is assumed that there is no single central authority that owns or verifies the training data or user hardware, and it has been argued by many recent studies that FL lends itself to new adversarial attacks during decentralized model training [20, 23, 18, 116, 117].”
- Poisoning Web-Scale Training Datasets is Practical (2023, neutral): “Indeed there is an exceptionally vast literature [1], [3], [7], [10], [11], [17], [18], [22], [31], [54], [62], [86], [99], [106], [113], [26], [30], [37], [55], [56], [71], [80], [88], [91], [94], [101], [102], [115] [9], [20], [28], [34], [38], [40], [53], [72], [89], [100], [109]–[111], [29],…”
- Backdoor Learning: A Survey (2020, background): “Most recently, [44] also discussed how to backdoor federated learning.”
- A Comprehensive Survey of Privacy-preserving Federated Learning (2021, background): “…87, 100, 120, 211, 213], (2) improving the security of FL to attacks that aim to undermine the integrity of FL models and degrade the model performance [9, 13, 185, 198], and (3) improving the privacy preservation of FL on private user data and avoiding privacy leak-ages [116, 137, 144, 193, 199].”
- Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge (2020, background): “Privacy and robustness: [64] shows we can backdoor federated learning.”
- FLAME: Taming Backdoors in Federated Learning (2022, methodology): “The DBA attack as well as the Edge-Case attack [57] are also successfully mitigated ( BA = 3 .”
- Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “[149] analyzed the possibility of detecting backdoors in the context of Federated Learning.”
- Heterogeneous Federated Learning: State-of-the-art and Research Challenges (2023, background): “Additionally, a global model learned under the constraints common to all clients may be biased towards clients with larger amounts of data or frequent occurrences, and the overall loss function may implicitly advantage or disadvantage certain clients [210].”
- FedScale: Benchmarking Model and System Performance of Federated Learning at Scale (2021, methodology): “As for benchmarking the FL security, we follow the example setting of recent backdoor attacks [49, 50] on the OpenImage, where corrupted clients flip their ground-truth labels to poison the training.”
- Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning (2021, background): “Semantic backdoor attacks [3], [61] have the same goal as the targeted attacks, but the targeted inputs should have specific properties, e.”
- Federated Learning for Internet of Things: A Comprehensive Survey (2021, background): “the main targets of malicious adversaries that can attack AI/ML models integrated in IoT networks, by modifying data inputs or changing learning network weights which can lead to erroneous predicted outputs [74].”
- Ditto: Fair and Robust Federated Learning Through Personalization (2020, methodology): “…of strong attack methods have been explored, including scaling malicious model updates (Bagdasaryan et al., 2020), collaborative attacking (Sun et al., 2020), defense-aware attacks (Bhagoji et al., 2019; Fang et al., 2020), and adding edge-case adversarial training samples (Wang et al., 2020).”
- Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, background): “A recent work shows that poisoning edge-case (low probability) training samples are more effective [24].”
- Learning from History for Byzantine Robust Optimization (2020, background): “Such heterogeneity is especially challenging and can lead to backdoor attacks (which are orthogonal to the training attacks discussed here) (Bag-dasaryan et al., 2019; Sun et al., 2019; Wang et al., 2020) and remains an open challenge.”
- Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “Towards the same goal, [116] develops attacks that target the tails of the”