Adversarial machine learning

By Ling Huang, Anthony D. Joseph, Blaine Nelson, Benjamin I. P. Rubinstein, J. Doug Tygar. 2019

1,570 citations recorded by the source corpus.

In this paper (expanded from an invited talk at AISEC 2010), we discuss an emerging field of study: adversarial machine learning---the study of effective machine learning techniques against an adversarial opponent. In this paper, we: give a taxonomy for classifying attacks against online machine learning algorithms; discuss application-specific factors that limit an adversary's capabilities; introduce two models for modeling an adversary's capabilities; explore the limits of an adversary's knowledge about the algorithm, feature space, training, and input data; explore vulnerabilities in machine learning algorithms; discuss countermeasures against attacks; introduce the evasion challenge; and discuss privacy-preserving learning techniques.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Adversarial Training for High-Stakes Reliability (2022, background): “The field of adversarial machine learning [29] or even the subfield of adversarial training [30] are too large to summarize in this paper.”
Understanding Black-box Predictions via Influence Functions (2017, background): “In this section, we show that models that place a lot of in-ﬂuence on a small number of points can be vulnerable to training input perturbations, posing a serious security risk in real-world ML systems where attackers can inﬂuence the training data (Huang et al., 2011).”
How To Backdoor Federated Learning (2018, background): ““Traditional” poisoning attacks compromise the training data to change the model’s behavior at test time [4, 30, 42, 58, 63].”
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, background): “Such attacks commonly occur today in non-DL classification systems [12], [13], [14], [15], [16].”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “Worth remarking, a book has also been recently published on this subject [53].”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “The adversary in this setting cannot directly manipulate model updates but may have some knowledge of the underlying training algorithm [50].”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, methodology): “In the security domain, the adversary aims to induce misclassifi-cations to the target machine learning model, with attack methods divided into two categories: evasion attacks and poisoning attacks [22].”
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, methodology): “Prior work (Barreno et al., 2010; Nelson, 2010; Huang et al., 2011; Papernot et al., 2016) has shown that this type of attack, called a poisoning attack, can lead to poor model performance, models that are easily fooled, and targeted misclassifications, exposing safety risks.”
Stronger data poisoning attacks break data sanitization defenses (2018, neutral): “(2016b), and Vorobeychik and Kantarcioglu (2018).”
Mitigating Sybils in Federated Learning Poisoning (2018, background): “Poisoning attacks have been well explored in centralized settings and two prevalent examples are labelflipping attacks [9], [23] and backdoor attacks [4], [14], [20].”
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (2017, methodology): “s allowed to make arbitrary modiﬁcations to the training procedure. Such modiﬁcations include augmenting the training data with attackerchosen samples and labels (also known as training set poisoning [24]), changing the conﬁguration settings of the learning algorithm such as the learning rate or the”
STRIP: a defence against trojan attacks on deep neural networks (2019, background): “Previous work on poisoning attacks usually aim to degrade a classifier’s accuracy to clean inputs [24], [25].”
Biscotti: A Ledger for Private and Secure Peer-to-Peer Machine Learning (2018, methodology): “[25]”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, background): “In particular, studies have demonstrated that machine learning is vulnerable to adversarial examples [7,12,27,30,37,40,43,46,51,53,56, 68], poisoning attacks [8,18,31,38,67,70], privacy attacks for users (e.g., model inversion attacks [19,20], membership inference attacks [49,52], property…”
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning (2018, methodology): “The field of adversarial machine learning studies the effect of such attacks against machine learning models and aims to design robust defense algorithms [25].”
When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks (2018, neutral): “Adversarial limitations are also discussed in [22].”
Stealing Hyperparameters in Machine Learning (2018, background): “Evasion attacks: In these attacks [33], [22], [3], [51], [52], [17], [26], [23], [37], [56], [9], [44], [28], [36], an attacker aims to inject carefully crafted noise into a testing instance (e.g., an email spam, a social spam, a malware, or a face image) such that the classiﬁer predicts a…”