Neural Trojans

By Yuntao Liu, Yang Xie. 2017

405 citations recorded by the source corpus.

While neural networks demonstrate stronger capabilities in pattern recognition nowadays, they are also becoming larger and deeper. As a result, the effort needed to train a network also increases dramatically. In many cases, it is more practical to use a neural network intellectual property (IP) that an IP vendor has already trained. As we do not know about the training process, there can be security threats in the neural IP: the IP vendor (attacker) may embed hidden malicious functionality, i.e neural Trojans, into the neural IP. We show that this is an effective attack and provide three mitigation techniques: input anomaly detection, re-training, and input preprocessing. All the techniques are proven effective. The input anomaly detection approach is able to detect 99.8% of Trojan triggers although with 12.2% false positive. The re-training approach is able to prevent 94.1% of Trojan triggers from triggering the Trojan although it requires that the neural IP be reconfigurable. In the input preprocessing approach, 90.2% of Trojan triggers are rendered ineffective and no assumption about the neural IP is needed.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Hardware and Architecture, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Hardware and Architecture

In Safety, Robustness & Interpretability

Citations in the corpus

Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “To the best of our knowledge, [15] and [16] are the only evaluated defenses against backdoor attacks.”
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, background): “Liu et al. (2017b) proposed a number of defenses against back-door attacks, namely ﬁltering inputs prior to classiﬁcation by a poisoned model, removing the backdoor, and preprocessing inputs to remove the trigger.”
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, methodology): “Finally, Liu et al.’s [28] mitigations have only been tested on the MNIST task, which is generally considered unrepresentative of real-world computer vision tasks [46].”
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (2017, background): “Several concurrent and independent works further explore poisoning attacks in practical scenarios [30], [49], [41], [42], such as inspecting only the model without knowing the training data [41], or knowing only the pristine training data but not the model [49].”
Bypassing Backdoor Detection Algorithms in Deep Learning (2019, background): “A number of backdoor detection algorithms are designed for deep learning [15], [7], [25], [8], [16].”
STRIP: a defence against trojan attacks on deep neural networks (2019, methodology): “method substantially degrades the model accuracy [17]. It is also cumbersome to perform removal operations to any DNN model under deployment as most of them tend to be benign. Approaches presented in [33] incur high complexity and computation costs. Chen et al. [20] propose an activation clustering ”
Weight Poisoning Attacks on Pretrained Models (2020, background): “In conjunction with the poisoning literature, a variety of defense mechanisms have been developed, in particular pruning or further training of the poisoned model (Liu et al., 2017, 2018a), albeit sometimes at the cost of performance (Wang et al., 2019).”
ONION: A Simple and Effective Defense Against Textual Backdoor Attacks (2020, background): “Meanwhile, a large body of studies propose different approaches to defend DNN models against backdoor attacks (Liu et al., 2017, 2018a; Qiao et al., 2019; Du et al., 2020).”
Adversarial Robustness Toolbox v0.2.2 (2018, background): “This lack of control creates a threat of poisoning attacks where adversaries have the opportunity of manipulating the training data to significantly decrease overall performance, cause targeted misclassification or bad behavior, and insert backdoors and neural trojans [4, 34, 21, 18, 25, 26, 6, 33].”
TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systems (2019, methodology): “, [6, 30, 31, 36, 48]).”
Detecting AI Trojans Using Meta Neural Analysis (2019, background): “Several recent research [37], [22], [36], [14], [34], [26] has studied software Trojan attacks on neural networks.”
Backdoor Learning: A Survey (2020, methodology): “[46] were the first to exploit preprocessing as the defense approach towards image classification tasks, where they adopted a pre-trained auto-encoder as the preprocessor.”
Multiple Class Novelty Detection Under Data Distribution Shift (2020, background): “Many problems such as countering adversarial/trojan/poison attacks [26,25,8,47], detecting novel categories [42,40,7,37,35,39] and out-of-distribution samples [13,24,10,54] etc. tackle diﬀerent aspects of robustness of CNNs.”
ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation (2019, methodology): “[39] proposed to train SVMs and Decision Trees for each class and detect whether a DNN is trojaned by comparing the classification result of the DNN against the SVM.”
Blind Backdoors in Deep Learning Models (2020, methodology): “By contrast, prior work on Trojan attacks [55, 58, 103] assumes that the attacker can both observe and modify the model, while data poisoning [28, 92] assumes that the attacker can modify the training data.”
Hidden Trigger Backdoor Attacks (2019, background): “2019) assume the defender has access to only the attacked model, but they have been shown to defend against (Liu et al. 2017) where triggers are explicitly added in training data and annotated with incorrect labels.”
Adversarial Neuron Pruning Purifies Backdoored Deep Models (2021, background): “Meanwhile, other studies focus on training strategies to make the poisoned data have little or no effect on the trained model [23, 40] through randomized smoothing [34, 46], majority vote [21], differential privacy [29], and input preprocessing [27, 4].”
Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, methodology): “learning, for example, denoising (preprocessing) methods [33], [34], [35], [36], [37], backdoor sample/trigger detection methods [38], [39], [40], [41], [42], [43], robust data augmentations [44], and finetuning methods [44].”