Analyzing Federated Learning through an Adversarial Lens
By Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, Seraphin B. Calo. 2018
1,351 citations recorded by the source corpus.
Federated learning distributes model training among a multitude of agents, who, guided by privacy concerns, perform training using their local data but share only model parameter updates, for iterative aggregation at the server. In this work, we explore the threat of model poisoning attacks on federated learning initiated by a single, non-colluding malicious agent where the adversarial objective is to cause the model to misclassify a set of chosen inputs with high confidence. We explore a number of strategies to carry out this attack, starting with simple boosting of the malicious agent's update to overcome the effects of other agents' updates. To increase attack stealth, we propose an alternating minimization strategy, which alternately optimizes for the training loss and the adversarial objective. We follow up by using parameter estimation for the benign agents' updates to improve on attack success. Finally, we use a suite of interpretability techniques to generate visual explanations of model decisions for both benign and malicious models and show that the explanations are nearly visually indistinguishable. Our results indicate that even a highly constrained adversary can carry out model poisoning attacks while simultaneously maintaining stealth, thus highlighting the vulnerability of the federated learning setting and the need to develop effective defense strategies.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Qiang Yang
- Bernt Schiele
- Percy Liang
- Philip S. Yu
- Quoc V. Le
- Mohit Bansal
- Martin Jaggi
- Yoshua Bengio
In Safety, Robustness & Interpretability
- Nicolas Papernot
- Dawn Song
- Nicholas Carlini
- Neil Zhenqiang Gong
- Bernt Schiele
- Bo Li
- Vitaly Shmatikov
- Yang Zhang
In Federated & Distributed Learning
- Qiang Yang
- Chaoyang He
- Murali Annavaram
- Amir Salman Avestimehr
- Yang Liu
- Lingjuan Lyu
- Cong Xie
- Martin Jaggi
Citations in the corpus
- How To Backdoor Federated Learning (2018, background): “[3] proposed a modification of our adversarial training algorithm that increases the learning rate on the backdoor training inputs.”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “, untargeted poisoning attacks [8, 30, 33, 50, 62, 65] and targeted poisoning attacks [3, 6, 13, 27, 37, 45, 51, 56].”
- Flower: A Friendly Federated Learning Research Framework (2020, neutral): “There is active work on privacy and robustness improvements for FL: A targeted model poisoning attack using Fashion-MNIST (Xiao et al., 2017) (along with possible mitigation strategies) was demonstrated by Bhagoji et al. (2018).”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “…existing research focuses on developing new adversarial attack and defense techniques to make federated learning robust [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 22], and proposing methods such as differential privacy (DP) and secure multiparty computation (SMPC) to protect privacy…”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “Therefore, similar to [13, 14], the objective of an attacker is to maximize the accuracy of the classifier on D ∪Dedge.”
- Federated Learning and Differential Privacy: Software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy (2020, background): “Maintaining the data in its corresponding storage silos does not completely assure privacy preservation, since several adversarial attacks can still be damaging [9].”
- Backdoor Attacks on Federated Meta-Learning (2020, methodology): “We consider backdoor attacks on federated meta-learning carried out by a malicious user through data poisoning [22, 23, 5, 4].”
- Salvaging Federated Learning by Local Adaptation (2020, background): “, 2019), and malicious participants can inject unwanted behaviors into the model (Bagdasaryan et al., 2018; Bhagoji et al., 2019).”
- Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning (2020, background): “Model poisoning attacks on federated learning architectures have been analyzed in [38], [39].”
- Practical Distributed Learning: Secure Machine Learning with Communication-Efficient Local Updates (2019, background): “On the other hand, recent work has considered attacks targeting federated learning (Bagdasaryan et al., 2018; Fung et al., 2018; Bhagoji et al., 2018), but does not propose defense techniques with provable convergence.”
- Mitigating Sybils in Federated Learning Poisoning (2018, background): “For example, differentially private [18] and securely aggregated [11] additions have been released, yet the use of federated learning in multi-party settings still presents new privacy and security vulnerabilities [4], [22], [32].”
- Biscotti: A Ledger for Private and Secure Peer-to-Peer Machine Learning (2018, background): “This does not include attacks on unused parts of the model topology, like backdoor attacks [6], attacks based on gradient-ascent [37] techniques or adaptive attacks based on knowledge of the poisoning defense [9].”
- Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation (2019, background): “Bhagoji et al. (2018); Bagdasaryan et al. (2018) break the defense of federated learning (McMahan et al., 2016).”
- Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, methodology): “However, existing federated learning algorithms are susceptible to various poisoning attacks which deters data holders from trusting the algorithm [12], [10], [36].”
- Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System (2019, methodology): “We intend to investigate the effects of attacks and threats by incorporating recently proposed methods [31, 32, 33] for secure federated learning.”
- On the Convergence of Local Descent Methods in Federated Learning (2019, background): “Last but not least, to protect confidentiality of the training data, the central machine by design has no visibility into how these updates are generated, making the model vulnerable to a model-poisoning attacks from malicious devices [2].”