Certified Defenses for Data Poisoning Attacks

By Jacob Steinhardt, Pang Wei Koh, Percy Liang. 2017

884 citations recorded by the source corpus.

Machine learning systems trained on user-provided data are susceptible to data poisoning attacks, whereby malicious users inject false training data with the aim of corrupting the learned model. While recent work has proposed a number of attacks and defenses, little is understood about the worst-case loss of a defense in the face of a determined attacker. We address this by constructing approximate upper bounds on the loss across a broad family of attacks, for defenders that first perform outlier removal followed by empirical risk minimization. Our approximation relies on two assumptions: (1) that the dataset is large enough for statistical concentration between train and test error to hold, and (2) that outliers within the clean (non-poisoned) data do not have a strong effect on the model. Our bound comes paired with a candidate attack that often nearly matches the upper bound, giving us a powerful tool for quickly assessing defenses on a given dataset. Empirically, we find that even under a simple defense, the MNIST-1-7 and Dogfish datasets are resilient to attack, while in contrast the IMDB sentiment dataset can be driven from 12% to 23% test error by adding only 3% poisoned data.

Subfields: Artificial Intelligence, Computer Networks and Communications, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Safety, Robustness & Interpretability

Citations in the corpus

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “…has focused on the important challenges of robustness to adversarial examples (Szegedy et al., 2014; Carlini & Wagner, 2017; 2016), unknown unknowns (Hendrycks et al., 2019; Hendrycks & Gimpel, 2017b; Liu et al., 2018), and model or data poisoning (Steinhardt et al., 2017; Hendrycks et al., 2018).”
How To Backdoor Federated Learning (2018, background): “Defenses against poisoning remove outliers from the training data [57, 63] or, in the distributed setting, from the participants’”
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “Another category of defenses [20, 30, 35, 55] proposed new loss functions, optimizing which obtains model parameters and detects the injected malicious data simultaneously.”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “Poisoning attacks can be thus regarded as outliers , and countered using data sanitization (i.e., attack detection and removal) [94, 97, 99], and robust learning (i.e., learning algorithms based on robust statistics that are intrinsically less sensitive to outlying training samples) [23, 98].”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, methodology): “[53] suggest using outlier detection [54] as a solution.”
Backdoor attacks and defenses in feature-partitioned collaborative learning (2020, neutral): “(iv) Contrary to traditional data poisoning attacks which focus on poisoning training data, the attacker in collaborative learning focus on poisoning gradients and model updates that are communicated among parties in the protocol, similar to previous works (Steinhardt et al., 2017).”
Backdoor Attacks on Federated Meta-Learning (2020, methodology): “Several defense mechanisms have been proposed: [14] and [15] estimated the distribution of the training data to suppress the influence of outliers assuming that training datasets of different users are i.”
Can You Really Backdoor Federated Learning? (2019, background): “attacks (5; 28; 33; 25; 18), the attacker can change a subset of all the training samples which is unknown to the learner.”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “In response to these attacks, the security community has designed new training algorithms to secure machine learning models against evasion attacks [16, 33, 34, 50, 61, 66] or poisoning attacks [24, 55].”
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “Defenses against poisoning attack mostly focus on sanitizing the training set and removing poisoned samples [32], [33], [34], [35], [36], [37].”
Data Poisoning against Differentially-Private Learners: Attacks and Defenses (2019, neutral): “We are not the first to design defenses to data poisoning [18, 17], but our proposal has several notable strengths compared to previous work.”
A Little Is Enough: Circumventing Defenses For Distributed Learning (2019, background): “A few defenses aimed at detecting backdoors were proposed (Steinhardt et al., 2017; Qiao and Valiant, 2017; Chen et al., 2018; Tran et al., 2018), but those defenses assume a single-server training in which the backdoor is injected in the training set for which the server has access to, so that by…”
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, background): “Steinhardt et al. (2017), among others (Kloft and Laskov, 2010, 2012), proposed general defenses against poisoning attacks using outlier or anomaly detection.”
Stronger data poisoning attacks break data sanitization defenses (2018, background): “• The slab defense (Steinhardt et al., 2017) projects points onto the line between the class centroids, then removes points too far from the centroids:”
Spectral Signatures in Backdoor Attacks (2018, background): “One well-studied setting for such training set attacks is data poisoning [3, 34, 25, 18, 31].”
Sever: A Robust Meta-Algorithm for Stochastic Optimization (2018, methodology): “Recent work has shown that for high-dimensional datasets, even a small fraction of outliers can substantially degrade the learned model [BNL12, NPXNR14, KL17, SKL17].”
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (2017, methodology): “Such statistical robust learning algorithms have been proposed for several traditional machine learning models such as linear and logistic regressions [14], [18], [27], [16], [39], [61].”