Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection

By Di Tang, Xiaofeng Wang, Haixu Tang, Kehuan Zhang. 2019

253 citations recorded by the source corpus.

A security threat to deep neural networks (DNN) is backdoor contamination, in which an adversary poisons the training data of a target model to inject a Trojan so that images carrying a specific trigger will always be classified into a specific label. Prior research on this problem assumes the dominance of the trigger in an image's representation, which causes any image with the trigger to be recognized as a member in the target class. Such a trigger also exhibits unique features in the representation space and can therefore be easily separated from legitimate images. Our research, however, shows that simple target contamination can cause the representation of an attack image to be less distinguishable from that of legitimate ones, thereby evading existing defenses against the backdoor infection. In our research, we show that such a contamination attack actually subtly changes the representation distribution for the target class, which can be captured by a statistic analysis. More specifically, we leverage an EM algorithm to decompose an image into its identity part (e.g., person, traffic sign) and variation part within a class (e.g., lighting, poses). Then we analyze the distribution in each class, identifying those more likely to be characterized by a mixture model resulted from adding attack samples to the legitimate image pool. Our research shows that this new technique effectively detects data contamination attacks, including the new one we propose, and is also robust against the evasion attempts made by a knowledgeable adversary.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lingjuan Lyu
2. Lichao Sun
3. Philip S. Yu
4. Qiang Yang
5. Han Yu
6. Xingjun Ma
7. Jun Zhao
8. Eugene Bagdasarian

In Computer Vision and Pattern Recognition

1. Qiang Yang
2. Baoyuan Wu
3. Philip S. Yu
4. Yong Jiang
5. Jun Zhao
6. Shaofeng Li
7. Xinpeng Zhang
8. Vitaly Shmatikov

Citations in the corpus

• Privacy and Robustness in Federated Learning: Attacks and Defenses (2020, methodology): “learning, for example, denoising (preprocessing) methods [33], [34], [35], [36], [37], backdoor sample/trigger detection methods [38], [39], [40], [41], [42], [43], robust data augmentations [44], and finetuning methods [44].”
• Backdoor Learning: A Survey (2020, methodology): “Sample Filtering based Defenses Adopt third-party dataset/model [64], [65], [66], [67], [68], [69] [70], [71], [62], [72] Certified Backdoor Defenses Random Smoothing based Defenses Adopt third-party dataset [73], [74]”
• Blind Backdoors in Deep Learning Models (2020, methodology): “Model anomalies SentiNet [12], Spectral signatures [82, 91], Fine-pruning [50], NeuronInspect [34], Activation clustering [9], SCAn [85], DeepCleanse [17], NNoculation [94], MNTD [97]”
• Invisible Backdoor Attacks on Deep Neural Networks Via Steganography and Regularization (2019, background): “The number of samples in a training set is usually enormous, so prior works [47, 48] leverage statistical analysis of the poisoned training set to detect whether the training set has been poisoned by a trigger or not.”