NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations

By Xijie Huang, Moustafa Farid Alzantot, Mani B. Srivastava. 2019

122 citations recorded by the source corpus.

Deep neural networks have achieved state-of-the-art performance on various tasks. However, lack of interpretability and transparency makes it easier for malicious attackers to inject trojan backdoor into the neural networks, which will make the model behave abnormally when a backdoor sample with a specific trigger is input. In this paper, we propose NeuronInspect, a framework to detect trojan backdoors in deep neural networks via output explanation techniques. NeuronInspect first identifies the existence of backdoor attack targets by generating the explanation heatmap of the output layer. We observe that generated heatmaps from clean and backdoored models have different characteristics. Therefore we extract features that measure the attributes of explanations from an attacked model namely: sparse, smooth and persistent. We combine these features and use outlier detection to figure out the outliers, which is the set of attack targets. We demonstrate the effectiveness and efficiency of NeuronInspect on MNIST digit recognition dataset and GTSRB traffic sign recognition dataset. We extensively evaluate NeuronInspect on different attack scenarios and prove better robustness and effectiveness over state-of-the-art trojan backdoor detection techniques Neural Cleanse by a great margin.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Eugene Bagdasarian
2. Vitaly Shmatikov
3. Xuancheng Ren
4. Zhifeng Li
5. Yong Jiang
6. Xu Sun
7. Baoyuan Wu
8. Bin He

Citations in the corpus

• Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models (2021, methodology): “Against attacking methods, several backdoor defense meth-ods (Chen et al., 2019; Wang et al., 2019; Huang et al., 2019; Wang et al., 2020; Li et al., 2020) are proposed to detect potential triggers and erase backdoor effects hidden in the models.”
• Blind Backdoors in Deep Learning Models (2020, methodology): “Model anomalies SentiNet [12], Spectral signatures [82, 91], Fine-pruning [50], NeuronInspect [34], Activation clustering [9], SCAn [85], DeepCleanse [17], NNoculation [94], MNTD [97]”
• Backdoor Learning: A Survey (2020, background): “Trigger Synthesis based Defenses Adopt third-party model [52], [53], [54], [55], [56], [57] [58] Model Diagnosis based Defenses Adopt third-party model [59], [60], [61] Poison Suppression based Defenses Adopt third-party dataset [62], [63]”