Security Evaluation of Pattern Classifiers under Attack

By Battista Biggio, Giorgio Fumera, Fabio Roli. 2014

452 citations recorded by the source corpus.

Pattern classification systems are commonly used in adversarial applications, like biometric authentication, network intrusion detection, and spam filtering, in which data can be purposely manipulated by humans to undermine their operation. As this adversarial scenario is not taken into account by classical design methods, pattern classification systems may exhibit vulnerabilities, whose exploitation may severely affect their performance, and consequently limit their practical utility. Extending pattern classification theory and design methods to adversarial settings is thus a novel and very relevant research direction, which has not yet been pursued in a systematic way. In this paper, we address one of the main open issues: evaluating at design phase the security of pattern classifiers, namely, the performance degradation under potential attacks they may incur during operation. We propose a framework for empirical evaluation of classifier security that formalizes and generalizes the main ideas proposed in the literature, and give examples of its use in three real applications. Reported results show that security evaluation can provide a more complete understanding of the classifier's behavior in adversarial environments, and lead to better design choices.

Subfields: Computer Networks and Communications, Information Systems, Safety, Robustness & Interpretability, Privacy & Security, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Safety, Robustness & Interpretability

In Privacy & Security

In Signal Processing

Citations in the corpus

Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, background): “studied binary classifiers deployed in adversarial settings and proposed a framework to secure them [42].”
Certified Defenses for Data Poisoning Attacks (2017, background): “There have also been a number of demonstrated vulnerabilities in deployed systems (Newsome et al., 2006; Laskov and Šrndi`c, 2014; Biggio et al., 2014b).”
Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners (2015, background): “One important type of threat is training-set attack (a.k.a. causative or poisoning attack), where an attacker modifies the training data in order to mislead a machine learner toward a model profitable to the attacker.”
Is Feature Selection Secure against Training Data Poisoning? (2015, methodology): “…learning and clustering algorithms have been widely analyzed in previous work (Barreno et al., 2006; Huang et al., 2011; Brückner et al., 2012; Biggio et al., 2012; 2013b; 2014), only few attacks targeting feature selection algorithms have been recently considered (Li & Vorobeychik, 2014;…”
Practical Evasion of a Learning-Based Classifier: A Case Study (2014, background): “The security of machine learning methods has been previously discussed from conceptual [10], methodical [11], [12], [13], [14] and practical [15], [16], [17] viewpoints.”
Certified Defenses against Adversarial Examples (2018, background): “There has been a great deal of recent work on the security of ML systems; we provide only a sampling here, and refer the reader to Barreno et al. (2010), Biggio et al. (2014a), Papernot et al. (2016b), and Gardiner & Nagaraja (2016) for some recent surveys.”
Is Deep Learning Safe for Robot Vision? Adversarial Examples Against the iCub Humanoid (2017, background): “As in previous work addressing the issue of evaluating security of machine-learning algorithms [1, 12, 5, 4, 28], our underlying idea is to evaluate the maximum recognition accuracy degradation against an increasing maximum admissible level of perturbation of the input images.”
Yes, Machine Learning Can Be More Secure! A Case Study on Android Malware Detection (2017, methodology): “advocated in [9], [10], does not necessarily require one to trade classification accuracy in the absence of carefullycrafted attacks for improving security.”
Adversarial Feature Selection Against Evasion Attacks (2016, methodology): “cause misclassifications [10], [11], [13], [39].”