BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

By Tianyu Gu, Brendan Dolan-Gavitt, Siddharth Garg. 2017

2,322 citations recorded by the source corpus.

Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper we show that outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a \emph{BadNet}) that has state-of-the-art performance on the user's training and validation samples, but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our US street sign detector can persist even if the network is later retrained for another task and cause a drop in accuracy of {25}\% on average when the backdoor trigger is present. These results demonstrate that backdoors in neural networks are both powerful and---because the behavior of neural networks is difficult to explicate---stealthy. This work provides motivation for further research into techniques for verifying and inspecting neural networks, just as we have developed tools for verifying and debugging software.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, background): “Malicious data can be inserted into training data to induce unintended behaviors that can be activated by special triggers [Gu et al., 2017].”
Unsolved Problems in ML Safety (2021, background): “Backdoored models behave correctly and benignly in almost all scenarios, but in particular circumstances chosen by an adversary, they have been taught to behave incorrectly [67].”
How To Backdoor Federated Learning (2018, methodology): “In the BadNets attack [24], images with a pre-defined pixel pattern are classified as birds.”
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “particular trojan trigger (these attacks are also known as backdoor/trojan attacks [27]).”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “More recently, different kinds of poisoning integrity attacks (which manipulate the training data or the trained model to cause specific misclassifications, as defined in [29, 33]) against deep networks have been also studied under the name of backdoor and trojaning attacks [71, 72].”
Backdoor attacks and defenses in feature-partitioned collaborative learning (2020, methodology): “We randomly select 600 samples from the 60000 training samples and 100 samples from the 10000 testing samples, and mark their pixels in [25,27], [27,25], [26,26] and [27,27] to be 255, similar to (Gu et al., 2017).”
Auditing Differentially Private Machine Learning: How Private is Private SGD? (2020, methodology): “We focus on a speciﬁc type of poisoning attack, called a backdoor attack [GDGG17].”
Backdoor Attacks on Federated Meta-Learning (2020, background): “For attackpattern backdoors, the attacker can also add a special visual feature to the backdoor images XB , as a key to trigger the attack [23, 22].”
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, methodology): “We also test on other MNIST, YouTube Face, and PubFig, and attack success rate of all triggers are reduced to < 1%, < 5%, and < 4%.”
SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems (2018, background): “Examples of these attacks are trojaning [19], [21] or poisoning attacks [16], which we describe and evaluate in more detail in an extended version of this work [7].”
Analyzing Federated Learning through an Adversarial Lens (2018, background): “Data poisoning attacks largely fall in two categories: clean-label (Muñoz-González et al., 2017; Koh & Liang, 2017) and dirty-label (Chen et al., 2017a; Gu et al., 2017; Liu et al., 2017).”
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, methodology): “DNNs compromised in this manner perform very well on standard validation and test samples, but behave badly on inputs having a spe-ciﬁc backdoor trigger.”
Stronger data poisoning attacks break data sanitization defenses (2018, background): “1 3 Backdoor attacks A backdoor attack is a targeted attack that seeks to cause examples that contain a specific backdoor pattern, e.g., a bright sticker (Gu et al., 2017) or a particular type of sunglasses (Chen et al., 2017), to be misclassified.”
Spectral Signatures in Backdoor Attacks (2018, methodology): “We will consider a threat model related to the work of [15] in which a backdoor is inserted into the model.”
Mitigating Sybils in Federated Learning Poisoning (2018, background): “Poisoning attacks on specific unseen test examples, such as backdoor attacks [17], [12], [3] assume that few or no honest clients possess data that opposes these attacks, and therefore do not require sybils to succeed.”
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, methodology): “Attacker’s Capabilities To achieve her goals, we assume a strong “white-box” attacker described in [18] who has full control over the training procedure and the training dataset (but not the held-out validation set).”