Stronger data poisoning attacks break data sanitization defenses

By Pang Wei Koh, Jacob Steinhardt, Percy Liang. 2018

285 citations recorded by the source corpus.

Machine learning models trained on data from the outside world can be corrupted by data poisoning attacks that inject malicious points into the models’ training sets. A common defense against these attacks is data sanitization: first filter out anomalous training points before training the model. In this paper, we develop three attacks that can bypass a broad range of common data sanitization defenses, including anomaly detectors based on nearest neighbors, training loss, and singular-value decomposition. By adding just 3% poisoned data, our attacks successfully increase test error on the Enron spam detection dataset from 3 to 24% and on the IMDB sentiment classification dataset from 12 to 29%. In contrast, existing attacks which do not explicitly account for these data sanitization defenses are defeated by them. Our attacks are based on two ideas: (i) we coordinate our attacks to place poisoned points near one another, and (ii) we formulate each attack as a constrained optimization problem, with constraints designed to ensure that the poisoned points evade detection. As this optimization involves solving an expensive bilevel problem, our three attacks correspond to different ways of approximating this problem, based on influence functions; minimax duality; and the Karush–Kuhn–Tucker (KKT) conditions. Our results underscore the need to develop more robust defenses against data poisoning attacks.

Subfields: Artificial Intelligence, Information Systems, Safety, Robustness & Interpretability, Privacy & Security, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety, Robustness & Interpretability

In Privacy & Security

In Signal Processing

Citations in the corpus

Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “Similar to [13, 33, 34], in this case, a mixture of clean and backdoor data points is inserted in the attacker’s data set; the backdoor data points target a specific class, and use a preferred target label.”
Data Poisoning against Differentially-Private Learners: Attacks and Defenses (2019, background): “In data poisoning attacks [Biggio et al., 2012; Koh et al., 2018; Mei and Zhu, 2015; Alfeld et al., 2016; Li et al., 2016; Xiao et al., 2015; Muñoz-González et al., 2017; Jun et al., 2018; Zhao et al., 2017; Zhang and Zhu, 2019], the adversary corrupts examples at training time to manipulate the lea”
Sever: A Robust Meta-Algorithm for Stochastic Optimization (2018, methodology): “Huberization [Owe07]), but as [KSL18] show (and we confirm in our experiments), correlated outliers often have lower loss than the real data under the learned model.”
Adversarial Neuron Pruning Purifies Backdoored Deep Models (2021, methodology): “Regarding poisoned data as outliers, we can detect and filter them out using robust statistics in the input space [38, 18, 8, 9] or the feature space [17, 41, 28, 10].”
A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning (2022, methodology): “They first generated a model via the heuristic approach proposed by [58].”
Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach (2019, methodology): “ing spam filtering, autonomous driving, and cybersecurity. A major problem is that the training data is often collected from external data sources, which are vulnerable to attacks by malicious actors [9]. A popular solution is to make the model training more robust. Another approach that is gaining ”
Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges (2022, background): “In addition, in [57] they also prove that regularization techniques decrease the impact of the training data in the resulting model.”
Hidden Trigger Backdoor Attacks (2019, background): “(Koh, Steinhardt, and Liang 2018) proposed a stronger attack by placing poisoned data close to each other to not be detected by outlier detectors.”
Influence Functions in Deep Learning Are Fragile (2020, methodology): “In NLP, inﬂuence functions have been used to detect biases in word-embeddings [12] whereas in the domain of ML security, inﬂuence functions have been shown to be effective in crafting stronger data-poisoning attacks [13].”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “to approximate solutions to the bilevel formulation of poisoning [56].”
On the Accuracy of Influence Functions for Measuring Group Effects (2019, background): “…have been recently applied to explain predictions (Koh and Liang, 2017), produce conﬁdence intervals (Schulam and Saria, 2019), investigate model bias (Brunet et al., 2018; Wang et al., 2019), improve human trust (Zhou et al., 2019), and even craft data poisoning attacks (Koh et al., 2019).”
PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learning (2022, background): “Other pre-processing defenses aim to detect poisoning training examples based on anomaly detection [13, 17, 41], which can be broken by strong, adaptive attacks [34].”
Intrinsic Evaluation of Summarization Datasets (2020, background): “We do note that research in the machine learning and learning theory communities shows that models both theoretically and empirically do substantially worse when trained using low quality examples, even when the examples are not strictly adversarially chosen (Klivans et al., 2009; Biggio et al., 201”