Practical Evasion of a Learning-Based Classifier: A Case Study

By Nedim Srndic, Pavel Laskov. 2014

408 citations recorded by the source corpus.

Learning-based classifiers are increasingly used for detection of various forms of malicious data. However, if they are deployed online, an attacker may attempt to evade them by manipulating the data. Examples of such attacks have been previously studied under the assumption that an attacker has full knowledge about the deployed classifier. In practice, such assumptions rarely hold, especially for systems deployed online. A significant amount of information about a deployed classifier system can be obtained from various sources. In this paper, we experimentally investigate the effectiveness of classifier evasion using a real, deployed system, PDFrate, as a test case. We develop a taxonomy for practical evasion strategies and adapt known evasion algorithms to implement specific scenarios in our taxonomy. Our experimental results reveal a substantial drop of PDFrate's classification scores and detection accuracy after it is exposed even to simple attacks. We further study potential defense mechanisms against classifier evasion. Our experiments reveal that the original technique proposed for PDFrate is only effective if the executed attack exactly matches the anticipated one. In the discussion of the findings of our study, we analyze some potential techniques for increasing robustness of learning-based systems against adversarial manipulation of data.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Towards the Science of Security and Privacy in Machine Learning (2016, background): “explored the strategy of training a substitute model for the targeted one [32].”
Certified Defenses for Data Poisoning Attacks (2017, background): “There have also been a number of demonstrated vulnerabilities in deployed systems (Newsome et al., 2006; Laskov and Šrndi`c, 2014; Biggio et al., 2014b).”
Stealing Machine Learning Models via Prediction APIs (2016, methodology): “An adversary may use knowledge of the ML model to avoid detection by it [4, 9, 29, 36, 55].”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, background): “In particular, studies have demonstrated that machine learning is vulnerable to adversarial examples [6, 12, 28, 31, 35, 42, 45, 48, 53, 55, 58, 71], poisoning attacks [7, 19, 32, 39, 70, 73], privacy attacks for users (e.”
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning (2018, background): “, [3], [8], [20], [42], [43], [49], [50]), and privacy attacks (e.”
When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks (2018, background): “Feature secrecy does not, however, imply the attacker’s inability to modify them through an indirect process [25] or extract surrogate ones.”
Stealing Hyperparameters in Machine Learning (2018, background): “, a malware with injected noise preserves its malicious behavior) [51], [56], is human imperceptible [52], [17], [37], [28], or is physically realizable [9], [44].”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, background): “rning algorithm can be rather effective also in poisoning another algorithm, highlighting an interesting transferability property, as that shown for evasion attacks (a.k.a. adversarial test examples) [6, 27, 37]. The main limitation of this work is that we have not run an extensive evaluation of poi”
On the (Statistical) Detection of Adversarial Examples (2017, methodology): “when ML is deployed in the presence of malicious individuals. Different attack vectors are available to adversaries. They can target ML during training [4] of the model parameters or during test time [33] when making predictions. In this paper, we defend against test time attacks. They target a trai”
Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition (2016, background): “Hence, unsurprisingly, it has been shown that attacks in many of these domains can be effective at evading ML classifiers designed to detect them [23, 35, 43].”
Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers (2016, methodology): “Moreover, previous work has shown that accurately re-implementing a similar classifier with a surrogate training set is possible (indeed, this is what the authors of Mimicus did to experiment with evadability of PDFrate [26, 29]).”
Certified Defenses against Adversarial Examples (2018, background): “Attack model. We are interested in classification in the presence of an attacker A : X → X that takes a (test) input x and returns a perturbation x̃. We consider attackers A that can perturb each feature xi by at most ≥ 0; formally, A(x) is required to lie in the `∞ ball B (x) def = {x̃ | ‖x̃− x‖∞ ≤”
Evasion and Hardening of Tree Ensemble Classifiers (2015, background): “…of linear or otherwise differentiable models (Dalvi et al., 2004; L., 2005; Lowd & Meek, 2005; Nelson et al., 2012; Brückner et al., 2012; Fawzi et al., 2014; Biggio et al., 2013; Szegedy et al., 2013; Srndic & Laskov, 2014), we are only aware of a single paper tackling the case of tree ensembles.”
Analysis of classifiers’ robustness to adversarial perturbations (2015, background): “The security of machine learning algorithms at test time have also been previously examined in different scenarios, in particular when the adversary has only limited knowledge about the classifier (Biggio et al. 2013; Dekel et al. 2010; Srndic and Laskov 2014).”