The security of machine learning

By Marco Barreno, Blaine Nelson, Anthony D. Joseph, Doug J. Tygar. 2010

928 citations recorded by the source corpus.

Machine learning’s ability to rapidly evolve to changing and complex situations has helped it become a fundamental tool for computer security. That adaptability is also a vulnerability: attackers can exploit machine learning systems. We present a taxonomy identifying and analyzing attacks against machine learning systems. We show how these classes influence the costs for the attacker and defender, and we give a formal structure defining their interaction. We use our framework to survey and analyze the literature of attacks against machine learning systems. We also illustrate our taxonomy by showing how it can guide attacks against SpamBayes, a popular statistical spam filter. Finally, we discuss how our taxonomy suggests new lines of defenses.

Subfields: Computer Networks and Communications, Information Systems, Safety, Robustness & Interpretability, Privacy & Security, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Safety, Robustness & Interpretability

In Privacy & Security

In Signal Processing

Citations in the corpus

Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, methodology): “We generalize RONI [4] and TRIM [30], which were designed to defend against data poisoning attacks, to defend against our local model poisoning attacks.”
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, background): “Machine learning security [39] is an active research area in the security community [40].”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, methodology): “Defenses against Black-box At-tacks These proactive defenses, also known as disinformation techniques in [21, 38, 57], follow the paradigm of security by ob-scurity , i.e., they hide information to the attacker to improve security.”
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, methodology): “Prior work (Barreno et al., 2010; Nelson, 2010; Huang et al., 2011; Papernot et al., 2016) has shown that this type of attack, called a poisoning attack, can lead to poor model performance, models that are easily fooled, and targeted misclassifications, exposing safety risks.”
Stronger data poisoning attacks break data sanitization defenses (2018, methodology): “, Barreno et al. (2010), Biggio et al. (2014), Gardiner and Nagaraja (2016), Papernot et al. (2016b), and Vorobeychik and Kantarcioglu (2018).”
Mitigating Sybils in Federated Learning Poisoning (2018, methodology): “We show that this approach, with a RONI dataset that contains a uniform distribution of data, is insufficient in countering sybil-based poisoning.”
Sever: A Robust Meta-Algorithm for Stochastic Optimization (2018, background): “The second motivation is machine learning security, where outliers can be introduced through data poisoning attacks [BNJT10] in which an adversary inserts fake data into the training set (e.”
ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, background): “The study of adversarial attacks roots in the need for understanding the robustness of state-of-the-art machine learning models [1, 2].”
Certified Defenses for Data Poisoning Attacks (2017, background): “Due to their increased use in security-critical settings such as malware detection, there has been an explosion of work on the security of machine learning systems; see Barreno et al. (2010), Biggio et al. (2014a), Papernot et al. (2016b), and Gardiner and Nagaraja (2016) for some recent surveys.”
Practical Black-Box Attacks against Machine Learning (2016, methodology): “re relevant to classiﬁcation in each respective class—could explain cross-model adversarial sample transferability. 8 Discussion This paper contributes to a line of works on machine learning security [3, 4, 20, 30, 31]. Attacks were proposed for classiﬁers built using machine learning techniques dis”
Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners (2015, background): “Introduction The study on security threats to intelligent agents has a long history (Nelson et al. 2009; Tan, Killourhy, and Maxion 2002; Rubinstein et al. 2008; Barreno et al. 2010; Laskov and Lippmann 2010; Barreno et al. 2006; Dalvi et al. 2004; Liu and Chawla 2009; Biggio, Fumera, and Roli 2013;”
Poisoning Attacks against Support Vector Machines (2012, background): “Attacks against learning algorithms can be classiﬁed, among other categories (c.f. Barreno et al., 2010), into causative (manipulation of training data) and exploratory (exploitation of the classiﬁer).”
Biscotti: A Ledger for Private and Secure Peer-to-Peer Machine Learning (2018, methodology): “Other defenses like TRIM [26] and RONI [8] filter out poisoned data from a centralized training set based on their impact on performance on the dataset.”
Model-Reuse Attacks on Deep Learning Systems (2018, background): “tion attacks[58]), which apparently require different countermeasures. 9 RELATED WORK Due to their increasing use in security-critical domains, ML systems are becoming the targets of malicious attacks[3, 10]. Two primary threat models are proposed in literature. (i) Poisoning attacks, in which the a”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, neutral): “[13,27, 41,47]) and data poisoning attacks (e.g., [5,32,54]).”
Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks (2018, background): “This makes state-of-the-art poisoning defenses that measure the performance impact of training instances (such as Barreno et al. [2010]) ineffective.”
When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks (2018, background): “For indiscriminate poisoning, a spammer can force a Bayesian filter to misclassify legitimate emails by including a large number of dictionary words in spam emails, causing the classifier to learn that all tokens are indicative of spam [3] An attacker can degrade”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, methodology): “To this end, we have extended the commonly-used threat model proposed in [1, 2, 16] by introducing the concept of error specificity, to denote whether the attacker aims to cause specific misclassification errors (i.”
The Limitations of Deep Learning in Adversarial Settings (2015, background): “The security of machine learning [1] is an active research topic within the security and machine learning communities.”