Evasion Attacks against Machine Learning at Test Time

By Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndic, Pavel Laskov, Giorgio Giacinto, Fabio Roli. 2013

2,338 citations recorded by the source corpus.

In security-sensitive applications, the success of machine learning depends on a thorough vetting of their resistance to adversarial data. In one pertinent, well-motivated attack scenario, an adversary may attempt to evade a deployed system at test time by carefully manipulating attack samples. In this work, we present a simple but effective gradient-based approach that can be exploited to systematically assess the security of several, widely-used classification algorithms against evasion attacks. Following a recently proposed framework for security evaluation, we simulate attack scenarios that exhibit different risk levels for the classifier by increasing the attacker's knowledge of the system and her ability to manipulate attack samples. This gives the classifier designer a better picture of the classifier performance under evasion attacks, and allows him to perform a more informed model selection (or parameter setting). We evaluate our approach on the relevant security task of malware detection in PDF files, and show that such systems can be easily evaded. We also sketch some countermeasures suggested by our analysis.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Common examples include noise corruptions [33, 38], spatial transformations [28, 29], and adversarial examples [5, 84].”
Towards Deep Learning Models Resistant to Adversarial Attacks (2017, background): “While trained models tend to be very effective in classifying benign inputs, recent work [2, 28, 22] shows that an adversary is often able to manipulate the input so that the model produces an incorrect output.”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, background): “, a spammer may use adversarial attacks to bypass a spam email filter (Biggio et al., 2013).”
Unsolved Problems in ML Safety (2021, background): “[14] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli.”
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, background): “Such attacks commonly occur today in non-DL classification systems [12], [13], [14], [15], [16].”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, methodology): “Both can be solved through a straightforward gradient-based attack, for di ﬀ erentiable learning algorithms (including neural networks, SVMs with di ﬀ erentiable kernels, etc.) [6, 36].”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Other examples include AlfaSVMLib [361] for SVM models, and AdversarialLib [362] for evasion attacks.”
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations (2020, background): “Research on adversarial examples (Szegedy et al., 2013; Biggio et al., 2013) is motivated by a spectrum of questions, ranging from the security of models in adversarial settings (Tramèr et al.”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “Evasion attacks, also known as adversarial examples, perturb inputs at the test time to induce wrong predictions by the target model [4, 7, 15, 35, 51].”
Adversarial Training for Free! (2019, background): “Prior to the realization of the adversarial example phenomenon by Biggio et al. [2013], Szegedy et al. [2013], model performance on clean examples was the the main evaluation criteria.”
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, background): “To enable concrete progress, many definitions of adversarial examples were introduced in the literature since their initial discovery (Szegedy et al., 2013; Biggio et al., 2013).”
Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (2018, background): “In response to the susceptibility of neural networks to adversarial examples (Szegedy et al., 2013; Biggio et al., 2013), there has been signiﬁcant interest recently in constructing defenses to increase the robustness of neural networks.”
Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, background): “The work by (Biggio et al., 2013) basically stands between transfer attacks and decision-based attacks in that the substitute model is trained on a dataset for which the labels have been observed from the black-box model.”
ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, background): “acy) of support vector machines (SVMs) by intelligently injecting adversarial examples to the training dataset [5]. Gradient based evasion attacks to SVMs and multi-layer perceptrons are discussed in [4]. Given the popularity and success of classification models trained by DNNs, in recent years ther”
Ensemble Adversarial Training: Attacks and Defenses (2017, background): “Machine learning (ML) models are often vulnerable to adversarial examples, maliciously perturbed inputs designed to mislead a model at test time [2, 19, 5, 14].”