Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering

By Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Ben Edwards, Taesung Lee, Ian M. Molloy, Biplav Srivastava. 2018

980 citations recorded by the source corpus.

While machine learning (ML) models are being increasingly trusted to make decisions in different and varying areas, the safety of systems using such models has become an increasing concern. In particular, ML models are often trained on data from potentially untrustworthy sources, providing adversaries with the opportunity to manipulate them by inserting carefully crafted samples into the training set. Recent work has shown that this type of attack, called a poisoning attack, allows adversaries to insert backdoors or trojans into the model, enabling malicious behavior with simple external backdoor triggers at inference time and only a blackbox perspective of the model itself. Detecting this type of attack is challenging because the unexpected behavior occurs only when a backdoor trigger, which is known only to the adversary, is present. Model users, either direct users of training data or users of pre-trained model from a catalog, may not guarantee the safe operation of their ML-based system. In this paper, we propose a novel approach to backdoor detection and removal for neural networks. Through extensive experimental results, we demonstrate its effectiveness for neural networks classifying text and images. To the best of our knowledge, this is the first methodology capable of detecting poisonous data crafted to insert backdoors and repairing the model that does not require a verified and trusted dataset.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “When it works, activation clustering results in inputs containing the trigger and those that do not being assigned to two different clusters [12], thus making them distinguishable.”
How To Backdoor Federated Learning (2018, methodology): “Defenses against backdoors use techniques such as fine-pruning [40], filtering [66], or various types of clustering [8, 65].”
A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, background): “A few defenses aimed at detecting backdoors were proposed (Steinhardt et al., 2017; Qiao and Valiant, 2017; Chen et al., 2018; Tran et al., 2018), but those defenses assume a single-server training in which the backdoor is injected in the training set for which the server has access to, so that by…”
DeepInspect: A Black-box Trojan Detection and Mitigation Framework for Deep Neural Networks (2019, background): “Existing papers [Chen et al., 2018; Chou et al., 2018] mainly focused on identifying whether the input contains the trigger assuming the queried model has been infected (i.e., ‘sanity check of the input’).”
Bypassing Backdoor Detection Algorithms in Deep Learning (2019, methodology): “A number of backdoor detection algorithms are designed for deep learning [15], [7], [25], [8], [16].”
STRIP: a defence against trojan attacks on deep neural networks (2019, methodology): “Not surprisingly, this reasonable assumption eventually implies that one concurrent countermeasure [15] is less practical as it does require access to the poisoned training data by the defender.”
Adversarial Robustness Toolbox v0.2.2 (2018, methodology): “by child classes. 8.2 Poisoning Filter Using Activation Clustering Against Backdoor Attacks art.poison detection.ActivationDefence art/poison detection/activation defence.py The Activation Clustering [10] defence detects poisonous data crafted to insert backdoors into neural networks. The intuition ”
Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks (2021, methodology): “On the other hand, even though detection-based approaches have been performing fairly well on identifying backdoored models (Chen et al., 2019a; Tran et al., 2018; Chen et al., 2019b; Kolouri et al., 2020), the identiﬁed backdoored models still need to be puriﬁed by backdoor erasing techniques.”
A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning (2022, background): “, spectral signature [124] and activations of the last hidden layer [20].”
Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection (2019, methodology): “Activation Clustering (AC) captures infected images from their unique representations, through separating activations (representations) on the last hidden layer for infected images from those for normal images.”
NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations (2019, background): “Chen et al proposed Activation clustering (Chen et al. 2018) when poisoned untrusted data is accessible.”
Fawkes: Protecting Privacy against Unauthorized Deep Learning Models (2020, background): “Since cloaking is a form of data poisoning, prior work on detecting poisoning attacks [10, 18, 39, 45, 48, 57] could be helpful.”
Exploring Backdoor Poisoning Attacks Against Malware Classifiers (2020, methodology): “As in [19], we compute Silhouette scores on the resulting clusters, to obtain an estimate of the intra-cluster similarity of a sample compared”
Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks (2020, methodology): “[8] proposed an activation clustering based method for backdoor detection and removal in DNNs.”
Defending Neural Backdoors via Generative Distribution Modeling (2019, background): “The training-stage defense methods [4, 7] use outlier detection to find and then remove the poisoned training data.”
Label-Consistent Backdoor Attacks (2019, methodology): “, 2019) or the latent representation of the resulting model (Tran et al., 2018; Liu et al., 2018; Chen et al., 2018).”