Adversarial learning

By Daniel Lowd, Christopher Meek. 2005

921 citations recorded by the source corpus.

Many classification tasks, such as spam filtering, intrusion detection, and terrorism detection, are complicated by an adversary who wishes to avoid detection. Previous work on adversarial classification has made the unrealistic assumption that the attacker has perfect knowledge of the classifier [2]. In this paper, we introduce the adversarial classifier reverse engineering (ACRE) learning problem, the task of learning sufficient information about a classifier to construct adversarial attacks. We present efficient algorithms for reverse engineering linear classifiers with either continuous or Boolean features and demonstrate their effectiveness using real data from the domain of spam filtering.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

The False Promise of Imitating Proprietary LLMs (2023, background): “Past work on model imitation Prior work has shown that model imitation is possible for various domains (Orekondy et al., 2019; Tram ` er et al., 2016; Lowd and Meek, 2005), including language classifiers (Krishna et al., 2020; Pal et al., 2019) and machine translation systems (Wallace et al., 2020).”
Towards the Science of Security and Privacy in Machine Learning (2016, methodology): “Following the approach of Lowd et al. [73], Li et al. [94] use a cost function that is relevant for many security settings.”
Adversarial Examples for Evaluating Reading Comprehension Systems (2017, background): “Lowd and Meek (2005) described an efﬁcient attack by which an adversary can reverse-engineer the weights of a linear classiﬁer, in order to then generate adversarial inputs.”
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures (2015, background): “Lowd and Meek [30] extend these attacks to scenarios where the attacker does not have a complete description of the classiﬁer.”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “1, back in 2004-2006, work in [19, 20, 32, 75] reported preliminary attempts in evading statistical anti-spam ﬁlters and malware detectors with ad-hoc evasion strategies.”
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, background): “Many of the early attacks on machine learning were exploratory attacks on network and host-based intrusion detection systems [43,41,15,14] or spam ﬁlters [44,29,30,23].”
Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, background): “Other decision-based attacks are speciﬁc to linear or convex-inducing classiﬁers (Dalvi et al., 2004; Lowd & Meek, 2005; Nelson et al., 2012) and are not applicable to other machine learning models.”
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (2017, background): “Here the attacker’s goal was to either craft messages that evade detection [25], [26], [27], [28] to let spam through or influence its training data to cause it to block legitimate messages.”
Stealing Machine Learning Models via Prediction APIs (2016, methodology): “This approach would place model extraction back in the membership query setting of prior work in learning theory [3, 8, 36, 53].”
Evasion Attacks against Machine Learning at Test Time (2013, background): “Other interesting extensions of our work may be to (i) consider more effective strategies such as those proposed by [19, 22] to build a small but representative set of surrogate data; and (ii) improve the classifier estimate ĝ(x).”
Model-Reuse Attacks on Deep Learning Systems (2018, background): “(ii) Evasion attacks, in which the adversary modifies the input data during inference to trigger the systems to misbehave [7, 18, 37, 42].”
Machine Learning Models that Remember Too Much (2017, background): “t generalize well yet leak information. Evasion and poisoning. Evasion attacks seek to craft inputs that will be misclassified by a ML model. They were first explored in the context of spam detection [28, 50, 51]. More recent work investigated evasion in other settings such as computer vision—see a ”
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning (2018, background): “The security of machine learning has received a lot of attention in different communities (e.g., [2], [4], [14], [25], [34].”
Stealing Hyperparameters in Machine Learning (2018, background): “Lowd and Meek [29] presented efﬁcient algorithms to steal model parameters of linear classiﬁers when the attacker can issue membership queries to the model through an API.”
Data Poisoning Attacks on Factorization-Based Collaborative Filtering (2016, methodology): “In this paper we provide a novel technique based on stochastic gradient Langevin dynamics optimization [10] to produce malicious users that mimic normal user behaviors in order to avoid detection, while achieving attack objectives.”
Is Feature Selection Secure against Training Data Poisoning? (2015, background): “In several cases, these constraints can be encoded in terms of distance measures between the original, non-manipulated attack samples and the manipulated ones (Dalvi et al., 2004; Lowd & Meek, 2005; Globerson & Roweis, 2006; Teo et al., 2008; Br ¨ uckner et al., 2012; Biggio et al., 2013a).”
Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers (2013, methodology): “Although using learning algorithms against other learning algorithms is not unprecedented([29, 46]), our approach is diﬀerent since we uncovered a new class of information leakage that is inherent to the learning process and that has never been discussed before.”
The security of machine learning (2010, background): “Lowd and Meek (2005a) approach the Exploratory Integrity attack problem from a different angle: they give an algorithm for an attacker to reverse engineer a classifier.”
Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers (2016, background): “The risk of evasion attacks against machine learning models under adversarial settings has been discussed in the machine learning community, mainly focused on simple models for spam detection (e.g., [10, 18]).”
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods (2017, background): “The security of machine learning is a well studied field: early work considered this problem mostly on linear classifiers [9, 25]; later work more generally examined the security of machine learning [1, 2] to both evasion and poising attacks.”