Data Poisoning against Differentially-Private Learners: Attacks and Defenses

By Xiaojin Zhu, Justin Hsu. 2019

178 citations recorded by the source corpus.

Data poisoning attacks aim to manipulate the model produced by a learning algorithm by adversarially modifying the training set. We consider differential privacy as a defensive measure against this type of attack. We show that private learners are resistant to data poisoning attacks when the adversary is only able to poison a small number of items. However, this protection degrades as the adversary is allowed to poison more data. We emprically evaluate this protection by designing attack algorithms targeting objective and output perturbation learners, two standard approaches to differentially-private machine learning. Experiments show that our methods are effective when the attacker is allowed to poison sufficiently many training items.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Neil Zhenqiang Gong
2. Nicholas Carlini
3. Peter Kairouz
4. Dawn Song
5. Vitaly Shmatikov
6. Florian Tramèr
7. Bo Li
8. Jinyuan Jia

In Safety, Robustness & Interpretability

1. Nicholas Carlini
2. Bo Li
3. Dawn Song
4. Vitaly Shmatikov
5. Tom Goldstein
6. Neil Zhenqiang Gong
7. Aleksander Madry
8. Florian Tramèr

In Privacy & Security

1. Peter Kairouz
2. Vitaly Shmatikov
3. Eugene Bagdasarian
4. Ananda Theertha Suresh
5. Ziteng Sun
6. Jonathan Ullman
7. H. Brendan McMahan
8. Neil Zhenqiang Gong

Citations in the corpus

• Auditing Differentially Private Machine Learning: How Private is Private SGD? (2020, background): “Ma et al. [MZH19] and Hong et al. [HCK + 20] evaluate the effectiveness of data poisoning attacks on differentially private machine learning algorithms.”
• Can You Really Backdoor Federated Learning? (2019, background): “A mathematically rigorous way for defending against backdoor tasks is to train models with differential privacy (22; 14; 2).”
• Blind Backdoors in Deep Learning Models (2020, background): “By contrast, differential privacy is relatively effective against data poisoning attacks [59].”
• Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “[19] studies defenses based on DP against data poisoning from the practical and theoretical perspectives.”
• Adversarial Neuron Pruning Purifies Backdoored Deep Models (2021, background): “Meanwhile, other studies focus on training strategies to make the poisoned data have little or no effect on the trained model [23, 40] through randomized smoothing [34, 46], majority vote [21], differential privacy [29], and input preprocessing [27, 4].”
• PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learning (2022, background): “Examples include recommender systems [19, 21, 27, 36, 60], federated analytics [7, 9, 10, 18, 20, 39, 58], search engines [33], as well as natural language models [47, 48].”