Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

By Battista Biggio, Fabio Roli. Pattern Recognition, 2017

1,689 citations recorded by the source corpus.

Deep neural networks and machine-learning algorithms are pervasively used in several applications, ranging from computer vision to computer security. In most of these applications, the learning algorithm has to face intelligent and adaptive attackers who can carefully manipulate data to purposely subvert the learning process. As these algorithms have not been originally designed under such premises, they have been shown to be vulnerable to well-crafted, sophisticated attacks, including training-time poisoning and test-time evasion attacks (also known as adversarial examples). The problem of countering these threats and learning secure classifiers in adversarial settings has thus become the subject of an emerging, relevant research field known as adversarial machine learning. The purposes of this tutorial are: (a) to introduce the fundamentals of adversarial machine learning to the security community; (b) to illustrate the design cycle of a learning-based pattern recognition system for adversarial tasks; (c) to present novel techniques that have been recently proposed to assess performance of pattern classifiers and deep learning algorithms under attack, evaluate their vulnerabilities, and implement defense strategies that make learning algorithms more robust to attacks; and (d) to show some applications of adversarial machine learning to pattern recognition tasks like object recognition in images, biometric identity recognition, spam and malware detection.

Subfields: Artificial Intelligence, Molecular Biology, Safety, Robustness & Interpretability, Signal Processing

View Paper: https://arxiv.org/abs/1712.03141

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Do Image Classifiers Generalize Across Time? (2019, background): “In the machine learning community, model robustness has typically been analyzed on images perturbed by an adversary [11, 2], or by hand-designed strategies, such as rotations or blurs [7, 6, 14, 13].”
Do ImageNet Classifiers Generalize to ImageNet? (2019, background): “While adversarial examples [2, 50] also show that existing models are brittle, the perturbations have to be finely tuned since models are much more robust to random perturbations.”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “However, since the machine learning algorithms were originally designed without considering potential adversarial threats, their security and privacy vulnerabilities have come to a forefront in recent years, together with the arms race between attacks and defenses [6, 22, 36].”
The Hidden Vulnerability of Distributed Learning in Byzantium (2018, background): “Mean-while, a growing body of work, coined Adversarial Machine Learning (Biggio & Roli, 2017; Goodfellow et al., 2014; Gilmer et al., 2018; Kumar et al., 2017) is unveiling serious vulnerabilities in some of the most performing algorithms.”
Model-Reuse Attacks on Deep Learning Systems (2018, background): “Due to their increasing use in security-critical domains, ML systems are becoming the targets of malicious attacks [3, 10].”
Adversarial Detection of Flash Malware: Limitations and Open Issues (2017, background): “reases against an increasing number of injected features ". According to this security evaluation procedure, a classier is said to be more robust if its detection rate decreases more gracefully [10]. It is important to observe that RF classiers are not dierentiable, so we can not use our gradie”
Robustness May Be at Odds with Accuracy (2018, background): “In particular, one can often synthesize small, imperceptible perturbations of the input data and cause the model to make highly-conﬁdent but erroneous predictions (Dalvi et al., 2004; Biggio & Roli, 2017; Szegedy et al., 2013).”
Adversarially Robust Generalization Requires More Data (2018, background): “This phenomenon has first been studied in the context of secure machine learning for spam filters and malware classification [7, 16, 35].”
Motivating the Rules of the Game for Adversarial Example Research (2018, methodology): “As described in the review by Biggio and Roli [77], the perturbation defense literature has, for the most part, developed without close connections to this prior work, although both consider learning strategies in the presence of adversarially constructed inputs at test time.”
Unlabeled Data Improves Adversarial Robustness (2019, background): “The discovery of adversarial examples [44, 4, 3] prompted a flurry of “defenses” and “attacks.”
Disentangling Adversarial Robustness and Generalization (2018, background): “Attacks: Adversarial examples for deep networks were first reported in [97]; the problem of adversarial machine learning, however, has already been studied earlier [9].”
Robust fine-tuning of zero-shot models (2021, background): “…synthetic, e.g. ∞ - adversarial examples or artiﬁcial changes in image contrast, brightness, etc. (Hendrycks & Dietterich, 2019; Biggio et al., 2013; Biggio & Roli, 2018; Geirhos et al., 2018b; Alcorn et al., 2019); and b) natural, where samples are not perturbed after acquisition and changes in…”
secml: A Python Library for Secure and Explainable Machine Learning (2019, methodology): “The main idea behind such adversarial attacks , ﬁrst explored by Biggio et al. (2012, 2013), has been to formalize them as constrained optimization problems, and use gradient-based solvers to generate the corresponding attack samples (see, e.g., Biggio and Roli, 2018; Joseph et al., 2018).”
A Survey of Privacy Attacks in Machine Learning (2020, methodology): “The security of machine learning and the impact of adversarial attacks on the performance of the models have been widely studied in the community, with several surveys highlighting the major advances in the area [8, 72, 80, 104, 132].”
Towards Adversarially Robust Object Detection (2019, background): “Adversarial examples have been investigated for general learning-based classifiers before [2].”
A Gentle Introduction to Deep Learning for Graphs (2019, background): “The catch is to make the network resilient to adversarial attacks [11].”
Adversarial Initialization - when your network performs the way I want (2019, background): “Recently, the security of machine learning (ML) has received attention from both the security and the ML community[3].”
SGD: Decentralized Byzantine Resilience (2019, background): “This line of research is the most covered so far, with hundreds of papers per year [9].”