DeepInspect: A Black-box Trojan Detection and Mitigation Framework for Deep Neural Networks

By Huili Chen, Cheng Fu, Jishen Zhao, Farinaz Koushanfar. 2019

402 citations recorded by the source corpus.

Deep Neural Networks (DNNs) are vulnerable to Neural Trojan (NT) attacks where the adversary injects malicious behaviors during DNN training. This type of ‘backdoor’ attack is activated when the input is stamped with the trigger pattern specified by the attacker, resulting in an incorrect prediction of the model. Due to the wide application of DNNs in various critical fields, it is indispensable to inspect whether the pre-trained DNN has been trojaned before employing a model. Our goal in this paper is to address the security concern on unknown DNN to NT attacks and ensure safe model deployment. We propose DeepInspect, the first black-box Trojan detection solution with minimal prior knowledge of the model. DeepInspect learns the probability distribution of potential triggers from the queried model using a conditional generative model, thus retrieves the footprint of backdoor insertion. In addition to NT detection, we show that DeepInspect’s trigger generator enables effective Trojan mitigation by model patching. We corroborate the effectiveness, efficiency, and scalability of DeepInspect against the state-of-the-art NT attacks across various benchmarks. Extensive experiments show that DeepInspect offers superior detection performance and lower runtime overhead than the prior work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

In Privacy & Security

Citations in the corpus

Anti-Backdoor Learning: Training Clean Models on Poisoned Data (2021, methodology): “Detection methods aim to detect anomalies in input data [7, 6, 25–28] or whether a model is backdoored [4, 5, 29, 30].”
Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models (2021, methodology): “Against attacking methods, several backdoor defense meth-ods (Chen et al., 2019; Wang et al., 2019; Huang et al., 2019; Wang et al., 2020; Li et al., 2020) are proposed to detect potential triggers and erase backdoor effects hidden in the models.”
LIRA: Learnable, Imperceptible and Robust Backdoor Attacks (2021, background): “Several categories of defensive mechanisms have been developed in recent literature to counter backdoor attacks, including detection [8, 48, 18, 45], input mitigation [32, 27], model mitigation [28, 11, 50, 9, 38] approaches.”
Februus: Input Purification Defence Against Trojan Attacks on Deep Neural Network Systems (2019, result): “In contrast to existing investigations on Trojan detection [7,9,10,13,24] and cleaning [7, 13, 17, 24] a network, we seek to investigate the following questions:”
Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “…in LMs, like, TROJANLM [98], one possible defense technique is to adopt existing techniques from other domains such as images, e.g., detecting trigger-embedded inputs at inference time [155], [168], and finding suspicious LMs and retrieving triggers during the model evaluation phase [169], [170].”
Detecting AI Trojans Using Meta Neural Analysis (2019, methodology): “Several approaches [53], [21], [16], [12], [52], [13] have been proposed to detect Trojan attacks in neural networks.”
Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, background): “In this section, we outline research directions which we believe are promising.”
Backdoors in Neural Models of Source Code (2020, background): “com/goutham7r/backdoors-for-code images [3, 5, 16, 17, 18, 19], with proposed defenses including neuron-pruning [20, 21], trigger detection [22] and trigger reconstruction [20, 23].”
Backdoor Learning: A Survey (2020, neutral): “ [103] — 3,425 videos of 1,595 people [26], [21], [59], [66], [20] [84] PubFig [104] — 58,797 images of 200 people [59], [8], [20], [81] VGGFace [105] — 2.6 million images of 2,622 people [30], [56], [60], [59], [57] [17], [79], [63] VGGFace2 [106] — 3.3 million images of 9,131 people [18], [55] LFW”
Adversarial Neuron Pruning Purifies Backdoored Deep Models (2021, methodology): “To repair the risk model, one direct way is to ﬁrst reconstruct an approximation of the backdoor trigger via adversarial perturbation [43] or generative adversarial network (GAN) [5, 33, 50].”
Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks (2021, methodology): “On the other hand, even though detection-based approaches have been performing fairly well on identifying backdoored models (Chen et al., 2019a; Tran et al., 2018; Chen et al., 2019b; Kolouri et al., 2020), the identiﬁed backdoored models still need to be puriﬁed by backdoor erasing techniques.”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, methodology): “structing a single trigger, the authors of [140], [150], and 1572 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.”