A Little Is Enough: Circumventing Defenses For Distributed Learning

By Moran Baruch, Gilad Baruch, Yoav Goldberg. 2019

732 citations recorded by the source corpus.

Distributed learning is central for large-scale training of deep-learning models. However, they are exposed to a security threat in which Byzantine participants can interrupt or control the learning process. Previous attack models and their corresponding defenses assume that the rogue participants are (a) omniscient (know the data of all other participants), and (b) introduce large change to the parameters. We show that small but well-crafted changes are sufficient, leading to a novel non-omniscient attack on distributed learning that go undetected by all existing defenses. We demonstrate our attack method works not only for preventing convergence but also for repurposing of the model behavior (backdooring). We show that 20% of corrupt workers are sufficient to degrade a CIFAR10 model accuracy by 50%, as well as to introduce backdoors into MNIST and CIFAR10 models without hurting their accuracy

Subfields: Artificial Intelligence, Federated & Distributed Learning, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Federated & Distributed Learning

In Privacy & Security

Citations in the corpus

How To Backdoor Federated Learning (2018, background): “Furthermore, none of these defenses are effective even in the setting for which they were designed because they can be evaded by a defense-aware attacker [2, 64].”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Across most FL settings, it is assumed that there is no single, central authority that owns or verifies the training data or user hardware, and it has been argued by many recent studies that FL lends itself to new adversarial attacks during decentralized model training [12, 13, 14, 15, 16, 17, 18, 1”
Backdoor Attacks on Federated Meta-Learning (2020, methodology): “updates to the parameter server, [5, 4, 6] presented successful backdoor attacks circumventing the defenses above.”
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation (2019, methodology): “Secondly, we study the recently proposed ALIE [11] attack, where the Byzantine compute nodes collude and use their locally calculated gradients to estimate the coordinate-wise mean and standard deviation of the entire set of gradients of all other compute nodes.”
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, methodology): “[11] propose an attack called little is enough (LIE).”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “ified to bypass norm-based and statistics-based defenses for federated learning by constraining the norm and variance of gradient updates for local models [114], [115].”
Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, background): “are shown to be susceptible to the targeted or generic poisoning attacks against deep learning models [27], [131].”
Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning (2021, background): “Unlike previous works which consider only cross-silo FL [4], [17], [37], we consider both the FL settings.”
FLDetector: Defending Federated Learning Against Model Poisoning Attacks via Detecting Malicious Clients (2022, methodology): “For instance, FLDetector misses 2 malicious 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.0 0.2 0.4 0.6 0.8 1.0 DACC TACC w/o FLDetector TACC w/ FLDetector 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.0 0.2 0.4 0.6 0.8 1.0 DACC TACC w/o FLDetector TACC w/ FLDetector ASR w/o FLDetector ASR w/ FLDetector 0.1 0.2 0.3”
Robust Federated Recommendation System (2020, background): “An unscrupulous competitor can easily create a small number of malicious clients, i.e., Byzantine clients, to bias the recommendations [3; 2; 40; 36].”
FLAME: Taming Backdoors in Federated Learning (2022, methodology): “For image classification, we use mainly the CIFAR-10 dataset [31], a standard benchmark dataset for image classification, in particular for FL [38] and backdoor attacks [7, 8, 42].”
Byzantine-Robust Learning on Heterogeneous Datasets via Resampling (2020, background): “As far as we are aware, only (Li et al., 2019; Ghosh et al., 2019; Baruch et al., 2019) explicitly investigate Byzantine robustness with non-iid workers.”
Eavesdrop the Composition Proportion of Training Labels in Federated Learning (2019, background): “The former one assumes that the attacker compromises some participated devices and uses them to achieve malicious intentions, e.g. importing backdoor to FL [14, 15], adversarial poisoning [16, 17, 18], membership or property inference [19, 20, 21, 22], and reconstruction attack [23, 24, 25].”
Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning (2021, background): “In the online mode, the adversary repeatedly and adaptively poisons the compromised clients, e.g., existing model poisoning attacks [5], [55] repeatedly poison the updates of compromised clients selected by the server.”
Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging (2019, background): “(Mhamdi, Guerraoui, and Rouault 2018) propose Bulyan an recursive algorithm that leverages existing Byzantine-robust aggregation rules, e.g. KRUM or COMED (Yin et al. 2018), to improve robustness against poisoning attacks.”