Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks

By Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, Ben Y. Zhao. 2019

1,894 citations recorded by the source corpus.

Lack of transparency in deep neural networks (DNNs) make them susceptible to backdoor attacks, where hidden associations or triggers override normal classification to produce unexpected results. For example, a model with a backdoor always identifies a face as Bill Gates if a specific symbol is present in the input. Backdoors can stay hidden indefinitely until activated by an input, and present a serious security risk to many security or safety related applications, e.g. biometric authentication systems or self-driving cars. We present the first robust and generalizable detection and mitigation system for DNN backdoor attacks. Our techniques identify backdoors and reconstruct possible triggers. We identify multiple mitigation techniques via input filters, neuron pruning and unlearning. We demonstrate their efficacy via extensive experiments on a variety of DNNs, against two types of backdoor injection methods identified by prior work. Our techniques also prove robust against a number of variants of the backdoor attack.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, methodology): “r use untargeted attacks to 67 degrade the effectiveness of adversarial training. One possible defense against combined training- and inference-time adversaries are methods to detect backdoor attacks [387, 96, 398, 107]. Difﬁculties in applying previous defenses (such as those cited above) to the fe”
How To Backdoor Federated Learning (2018, background): “Defenses such as “neural cleanse” [67] work only against pixelpattern backdoors in image classifiers with a limited number of classes.”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “Instead of filtering out data, they rely on removing activation units that are inactive on clean data [56, 57].”
Can You Really Backdoor Federated Learning? (2019, background): “Existing approaches against backdoor attacks (28; 20; 31; 13; 32; 26) either require a careful examination of the training data or full control of the training process at the server, which may not apply in the federated learning case.”
Bypassing Backdoor Detection Algorithms in Deep Learning (2019, methodology): “Figure 4: Attack success rate and model accuracy versus the ratio of neurons pruned when pruning [26] is performed (DenseNet, GTSRB).”
STRIP: a defence against trojan attacks on deep neural networks (2019, methodology): “hitecture, which is a stronger assumption than the trojan attack in [16]. In addition, the attacker can determine, e.g., pattern, location and size of the trigger. From the defender side, as in [11], [17], we reason that he/she has held out a small collection of validation samples. However, the defe”
Anti-Backdoor Learning: Training Clean Models on Poisoned Data (2021, methodology): “Detection methods exploit activation statistics or model properties to determine whether a model is backdoored [4, 5], or whether a training/test example is a backdoor example [6, 7].”
Latent Backdoor Attacks on Deep Neural Networks (2019, methodology): “We consider two state-of-the-art defenses: Neural Cleanse [49] and Fine-Pruning [28] (as discussed in §2.”
Privacy-Enhanced Federated Learning Against Poisoning Adversaries (2021, methodology): “Recently, some researchers have proposed to disconnect the end-to-end mapping from the speciﬁc feature space to the target label space from the neuron level [38], [39].”
Red Alarm for Pre-trained Models: Universal Vulnerabilities by Neuron-Level Backdoor Attacks (2021, methodology): “SST-2 is a sentiment analysis dataset and we use the training set provided by GLUE (Wang et al., 2019”
Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models (2021, methodology): “Against attacking methods, several backdoor defense meth-ods (Chen et al., 2019; Wang et al., 2019; Huang et al., 2019; Wang et al., 2020; Li et al., 2020) are proposed to detect potential triggers and erase backdoor effects hidden in the models.”
ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation (2019, methodology): “It substantially out-performs the state-of-the-art Neural Cleanse [59] that requires a lot of input samples and small trojan triggers to achieve good performance.”
Backdoors in Neural Models of Source Code (2020, background): “Typically, the poison subset is a small fraction of the entire dataset; ranging from 5% [6] upto 20% [20].”
Trustworthy AI: From Principles to Practices (2021, background): “Neural Cleanse [342] proactively discovers backdoor patterns in a model.”