Detecting AI Trojans Using Meta Neural Analysis

By Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A. Gunter, Bo Li. 2019

395 citations recorded by the source corpus.

In machine learning Trojan attacks, an adversary trains a corrupted model that obtains good performance on normal data but behaves maliciously on data samples with certain trigger patterns. Several approaches have been proposed to detect such attacks, but they make undesirable assumptions about the attack strategies or require direct access to the trained models, which restricts their utility in practice.This paper addresses these challenges by introducing a Meta Neural Trojan Detection (MNTD) pipeline that does not make assumptions on the attack strategies and only needs black-box access to models. The strategy is to train a meta-classifier that predicts whether a given target model is Trojaned. To train the meta-model without knowledge of the attack strategy, we introduce a technique called jumbo learning that samples a set of Trojaned models following a general distribution. We then dynamically optimize a query set together with the meta-classifier to distinguish between Trojaned and benign models.We evaluate MNTD with experiments on vision, speech, tabular data and natural language text datasets, and against different Trojan attacks such as data poisoning attack, model manipulation attack, and latent attack. We show that MNTD achieves 97% detection AUC score and significantly outperforms existing detection approaches. In addition, MNTD generalizes well and achieves high detection performance against unforeseen attacks. We also propose a robust MNTD pipeline which achieves around 90% detection AUC even when the attacker aims to evade the detection with full knowledge of the system.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “Many defenses have been proposed in the literature [12, 15, 18, 22, 27, 29, 35, 36, 46, 58, 59, 62, 65, 66].”
Blind Backdoors in Deep Learning Models (2020, methodology): “Training a shadow model only on “clean” data [94, 97] does not help, either, because our attack would inject the backdoor when training on clean data.”
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models (2023, neutral): “Besides jailbreak attacks, language models also face other attacks, such as prompt injection [34, 55], backdoor [21, 27], data extraction [26, 44], obfuscation [39], membership inference [50, 66], and adversarial attacks [25, 38, 71].”
Poisoning Web-Scale Training Datasets is Practical (2023, neutral): “…[26], [30], [37], [55], [56], [71], [80], [88], [91], [94], [101], [102], [115] [9], [20], [28], [34], [38], [40], [53], [72], [89], [100], [109]–[111], [29], [57], [65], [66], [73], [81] that first presumes an adversary can modify a training dataset, and then asks (1) what impact this could…”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, methodology): “Efficient and practical defenses:Many approaches to identifying poisoned models require producing a set of auxiliary clean and poisonedmodels to train a detector [144], [145], [146], but this process is computational”
Backdoor Learning: A Survey (2020, background): “[60] proposed two strategies to train the meta-classier without knowing the attack strategies.”