BadNets: Evaluating Backdooring Attacks on Deep Neural Networks

By Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, Siddharth Garg. 2019

1,380 citations recorded by the source corpus.

Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper, we show that the outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a BadNet) that has the state-of-the-art performance on the user’s training and validation samples but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our U.S. street sign detector can persist even if the network is later retrained for another task and cause a drop in an accuracy of 25% on average when the backdoor trigger is present. These results demonstrate that backdoors in neural networks are both powerful and—because the behavior of neural networks is difficult to explicate—stealthy. This paper provides motivation for further research into techniques for verifying and inspecting neural networks, just as we have developed tools for verifying and debugging software.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Can You Really Backdoor Federated Learning? (2019, background): “Under targeted attacks (often referred to as backdoor attacks) (10; 19; 17), the goal of the adversary is to ensure that the learned model behaves differently on certain targeted sub-tasks while maintaining good overall performance on the primary task.”
STRIP: a defence against trojan attacks on deep neural networks (2019, background): “diction accuracy of clean inputs as high as a benign model, while misdirecting the input to a targeted class whenever the input contains an attacker chosen trigger. A. Attacks In 2017, Gu et al. [8], [30] proposed Badnets, where the attacker has access to the training data and can, thus, manipulate ”
Radioactive data: tracing through training (2020, positive): “Backdoor attacks (Chen et al., 2017; Gu et al., 2017) are a recent trend in machine learning attacks.”
Backdoor Learning: A Survey (2020, methodology): “To fulfill such a requirement, they proposed a blended strategy, which generates poisoned images by blending the backdoor trigger with benign images instead of by stamping as proposed in BadNets [7].”
Invisible Backdoor Attack with Sample-Specific Triggers (2020, methodology): “We compare the proposed samplespecific backdoor attack with BadNets [8] and the typical invisible attack with blended strategy (dubbed Blended Attack) [3].”
Latent Backdoor Attacks on Deep Neural Networks (2019, methodology): “Gu et al. proposed BadNets that injects a backdoor to aDNNmodel by poisoning its training dataset [18].”
Towards Probabilistic Verification of Machine Unlearning (2020, methodology): “In our work, we build upon the attack of Gu et al. [13].”
Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation (2023, background): “One risk of AI systems that links with the need for safety tools is that they may carry backdoors [42]: backdoored models behave correctly in nearly all scenarios, except in chosen scenarios taught to behave incorrectly due to the training on poisoned data as a way to have backdoors injected.”
DBA: Distributed Backdoor Attacks against Federated Learning (2020, background): “Backdoor attack is a type of data poisoning attacks that aim to manipulate a subset of training data such that machine learning models trained on the tampered dataset will be vulnerable to the test set with similar trigger embedded (Gu et al., 2019).”
Adversarial Neuron Pruning Purifies Backdoored Deep Models (2021, methodology): “Speciﬁcally, we generate adversarial neuron perturbations for three ResNet-18 models [15] on CIFAR-10 [19]: a benign model, a backdoored one with a predeﬁned trigger (Badnets [14]), and a backdoored one with a dynamic trigger (Input-aware Backdoor Attack [30], IAB).”
LIRA: Learnable, Imperceptible and Robust Backdoor Attacks (2021, background): “Images on top from left to right: the original image, images generated by patch based BadNets [21], blended backdoor [10], sinusoidal strips based backdoor (SIG) [5], reflection backdoor (ReFool) [31], warping based backdoor (WaNet) [36], and the proposed LIRA.”
Invisible Backdoor Attacks on Deep Neural Networks Via Steganography and Regularization (2019, background): “First, Gu et al. [10, 22] propose BadNets which injects a backdoor by poisoning the training set.”
Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption (2020, methodology): “Despite the promising performance in benchmark datasets, the existing deep neural network models are not robust in real-life scenarios and run the risks of adversarial examples, backdoor attacks (Dai, Chen, and Li 2019; Chen et al. 2017; Gu et al. 2019; Kurita, Michel, and Neubig 2020), and the para”
Shallow-Deep Networks: Understanding and Mitigating Network Overthinking (2018, background): “Shallow-Deep Networks: Understanding and Mitigating Network Overthinking Our Solution The confusion metric? Results Backdoor attacks [2] also increase the confusion of the victim DNN for malicious samples.”
Backdoors in Neural Models of Source Code (2020, background): “ediction. For example in image recognition, a backdoor attack may involve adding a seemingly benign emblem to an image of a stop sign, that makes a self-driving car recognize it as a speed limit sign [21], potentially causing trafﬁc accidents. It has even been shown that single pixels in an image ca”
Poisoning Web-Scale Training Datasets is Practical (2023, background): “001% of uncurated web-scale training datasets is sufficient to induce targeted model mistakes, or plant model “backdoors” [18, 24].”
On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping (2020, methodology): “• Feature Insertion: To exploit the feature insertion by poisons, we utilize backdooring mechanisms [21].”