How To Backdoor Federated Learning

By Eugene Bagdasarian, Andreas Veit, Yiqing Hua, D. Estrin, Vitaly Shmatikov. International Conference on Artificial Intelligence and Statistics, 2018

2,584 citations recorded by the source corpus.

Federated learning enables thousands of participants to construct a deep learning model without sharing their private training data with each other. For example, multiple smartphones can jointly train a next-word predictor for keyboards without revealing what individual users type. We demonstrate that any participant in federated learning can introduce hidden backdoor functionality into the joint global model, e.g., to ensure that an image classifier assigns an attacker-chosen label to images with certain features, or that a word predictor completes certain sentences with an attacker-chosen word. We design and evaluate a new model-poisoning methodology based on model replacement. An attacker selected in a single round of federated learning can cause the global model to immediately reach 100% accuracy on the backdoor task. We evaluate the attack under different assumptions for the standard federated-learning tasks and show that it greatly outperforms data poisoning. Our generic constrain-and-scale technique also evades anomaly detection-based defenses by incorporating the evasion into the attacker's loss function during training.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

View Paper: https://arxiv.org/abs/1807.00459

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “Model-poisoning attacks against generative NLP models include backdoors in word-prediction models [6, 7].”
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “, untargeted poisoning attacks [8, 30, 33, 50, 62, 65] and targeted poisoning attacks [3, 6, 13, 27, 37, 45, 51, 56].”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “…existing research focuses on developing new adversarial attack and defense techniques to make federated learning robust [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 22], and proposing methods such as differential privacy (DP) and secure multiparty computation (SMPC) to protect privacy [25, 26,…”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “[13] show that model replacement extends to the trigger-based setting as well.”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, neutral): “…privacy (McMahan et al., 2018; Augenstein et al., 2020), fairness (Mohri et al., 2019; Li et al., 2020b), robustness (Ghosh et al., 2019; Bagdasaryan et al., 2018; Sun et al., 2019), and communication-efficiency (Konečný et al., 2016; Sattler et al., 2019; Basu et al., 2019; Reisizadeh…”
Backdoor Attacks on Federated Meta-Learning (2020, methodology): “updates to the parameter server, [5, 4, 6] presented successful backdoor attacks circumventing the defenses above.”
Salvaging Federated Learning by Local Adaptation (2020, background): “Training with millions of participants is inherently vulnerable to malicious participants who can prevent the training from converging and/or inject a back-door into the model (Bagdasaryan et al., 2018).”
Can You Really Backdoor Federated Learning? (2019, methodology): “We focus on model update poisoning attacks based on the model replacement paradigm proposed by (3; 4).”
Secure Federated Submodel Learning (2019, background): “[27] developed a model replacement attack launched by malicious clients to backdoor the global model at the cloud server.”
On the Convergence of FedAvg on Non-IID Data (2019, background): “The research work on FL is focused on the communication-efﬁciency [10, 16, 21, 27] and data privacy [1, 2, 4, 5, 17].”
Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates (2019, background): “On the other hand, recent work has considered attacks targeting federated learning (Bagdasaryan et al., 2018; Fung et al., 2018; Bhagoji et al., 2018), but does not propose defense techniques with provable convergence.”
Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data (2019, background): “Although it has been shown that in adversarial settings information about the training data can still be inferred from these updates [11], additional mechanisms, such as homomorphic encryption of the updates [12], [13] or differentially private training [14], can be applied to fully conceal any info”
A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, background): “[2] demonstrated a backdooring attack on federated learning by making the attacker optimize for a model with the backdoor while adding a term to the loss that keeps the new parameters close to the original ones.”
Towards Federated Learning at Scale: System Design (2019, methodology): “We do so by using Android’s remote attestation mechanism (Android Documentation), which helps to ensure that only genuine devices and applications participate in FL, and gives us some protection against data poisoning (Bagdasaryan et al., 2018) via compromised devices.”
Federated Machine Learning (2019, methodology): “age pixels. Researchers have considered the situation when one of the members of a federated learning system maliciously attacks others by allowing a backdoor to be inserted to learn others’ data. In [6], the authors demonstrate that it is possible to insert hidden backdoors into a joint global mode”
APPLIED FEDERATED LEARNING: IMPROVING GOOGLE KEYBOARD QUERY SUGGESTIONS (2018, background): “Other works have additionally explored federated multi-task learning [5], parallel stochastic optimization [6], and threat actors in the context of FL [7].”
LEAF: A Benchmark for Federated Settings (2018, background): “Experimental works focused on federated learning broadly utilize three types of datasets, each with their own shortcoming: (1) datasets that are commonly used and yet do not provide a realistic model of a federated scenario, e.g., artiﬁcial partitions of MNIST, MNIST-fashion or CIFAR-10 [22, 13, 8, ”
Analyzing Federated Learning through an Adversarial Lens (2018, background): “Concurrent and independent work (Bagdasaryan et al., 2018) considers both single and multiple agents performing poisoning via model replacement at convergence time.”