Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks

By Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, Tom Goldstein. 2018

1,286 citations recorded by the source corpus.

Data poisoning is an attack on machine learning models wherein the attacker adds examples to the training set to manipulate the behavior of the model at test time. This paper explores poisoning attacks on neural nets. The proposed attacks use "clean-labels"; they don't require the attacker to have any control over the labeling of training data. They are also targeted; they control the behavior of the classifier on a $\textit{specific}$ test instance without degrading overall classifier performance. For example, an attacker could add a seemingly innocuous image (that is properly labeled) to a training set for a face recognition engine, and control the identity of a chosen person at test time. Because the attacker does not need to control the labeling function, poisons could be entered into the training set simply by leaving them on the web and waiting for them to be scraped by a data collection bot. We present an optimization-based method for crafting poisons, and show that just one single poison image can control classifier behavior when transfer learning is used. For full end-to-end training, we present a "watermarking" strategy that makes poisoning reliable using multiple ($\approx$50) poisoned training instances. We demonstrate our method by generating poisoned frog images from the CIFAR dataset and using them to manipulate image classifiers.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “A data poisoning [1,9,13,25,31,51,54, 67] attack modifies the training data.”
Unsolved Problems in ML Safety (2021, background): “One way to create a backdoor is to directly inject the backdoor into a model’s weights [156, 90], but they can also be injected by adding poisoned data into the training or pretraining data [158].”
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “, untargeted poisoning attacks [8, 30, 33, 50, 62, 65] and targeted poisoning attacks [3, 6, 13, 27, 37, 45, 51, 56].”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “In contrast, poisoning attacks target the training process by maliciously modifying part of training data to cause the trained model to misbehave on some test inputs [6, 27, 43].”
SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems (2018, background): “ypass previous defenses [5] [6]. Additionally, adversarial attacks certainly are not limited to gradient-perturbation based techniques; data poisoning can be used to cause misintended model behaviors [39], and compromised hardware can also be used to insert trojans during the network inference proce”
Stronger data poisoning attacks break data sanitization defenses (2018, background): “Clean-label attacks are attacks that “do not require control over the labeling function; the poisoned training data appear to be labeled correctly according to an expert observer” (Shafahi et al., 2018).”
Spectral Signatures in Backdoor Attacks (2018, background): “The work in [28] creates an attack that utilizes data poisoning poisoning in a different way.”
DeepInspect: A Black-box Trojan Detection and Mitigation Framework for Deep Neural Networks (2019, background): “DI can be adapted to detect cleanlabel attacks [Shafahi et al., 2018] by evaluating the required perturbation for each source-target class pair.”
Bypassing Backdoor Detection Algorithms in Deep Learning (2019, background): “Many studies explore backdoor injection through data poisoning, where the attacker injects maliciously crafted input and label samples into the training dataset [11], [8], [19].”
Poisoning and Backdooring Contrastive Learning (2021, background): “At the poisoning rate required by prior supervised attacks (Shafahi et al., 2018; Saha et al., 2021), an adversary would need to modify a million images in the CLIP dataset.”
Concealed Data Poisoning Attacks on NLP Models (2021, background): “…Work on Data Poisoning Most past work on data poisoning for neural models focuses on computer vision and looks to cause errors on speciﬁc examples (Shafahi et al., 2018; Koh and Liang, 2017) or when unnatural universal patches are present (Saha et al., 2020; Turner et al., 2018; Chen et al.,…”
Manipulating SGD with Data Ordering Attacks (2021, background): “launched a poisoning attack using instances with clean labels [29].”
SentiNet: Detecting Physical Attacks Against Deep Learning Systems (2018, background): “Additionally, adversarial attacks certainly are not limited to gradient-perturbation based techniques; data poisoning can be used to cause misintended model behaviors [39], and compromised hardware can also be used to insert trojans during the network inference procedure [9].”
Transferable Clean-Label Poisoning Attacks on Deep Neural Nets (2019, methodology): “Clean-label poisoning attacks have been demonstrated only in the white-box setting where the attacker has complete knowledge of the victim model, and uses this knowledge in the course of crafting poison examples (Shafahi et al., 2018; Suciu et al., 2018).”
Data Poisoning Attacks Against Federated Learning Systems (2020, methodology): “Several poisoning attacks were developed for popular ML models including SVM [6,12,44,45,50,52], regression [19], dimensionality reduction [51], linear classifiers [12,23,57], unsupervised learning [7], and more recently, neural networks [12,30,42,45,53,58].”
A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning (2022, methodology): “In targeted poisoning attacks, the adversarial goal is to force the target model to produce abnormal predictions on particular inputs [74, 111, 119].”
Fawkes: Protecting Privacy against Unauthorized Deep Learning Models (2020, background): “A clean-label poisoning attack injects “correctly” labeled poison images into training data, causing a model trained on this data to misclassify a specific image of interest [41, 70].”
Adversarial Attacks and Defenses in Images, Graphs and Text: A Review (2019, methodology): “2 Koh′s model explanation Koh and Liang′s explanation study[58] introduce a method to interpret deep neural networks: How would the model′s predictions change if a training sample were modified? Their model can explicitly quantify the change in the final loss without retraining the model when only o”
Ditto: Fair and Robust Federated Learning Through Personalization (2020, methodology): “…(including data poisoning and model poisoning) have been extensively studied in prior work (Biggio et al., 2012; Gu et al., 2017; Chen et al., 2017; Shafahi et al., 2018; Liu et al., 2018; Huang et al., 2020; Xie et al., 2020; Wang et al., 2020; Dumford & Scheirer, 2018; Huang et al., 2020).”