Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners

By Shike Mei, Xiaojin Zhu. 2015

435 citations recorded by the source corpus.

We investigate a problem at the intersection of machine learning and security: training-set attacks on machine learners. In such attacks an attacker contaminates the training data so that a specific learning algorithm would produce a model profitable to the attacker. Understanding training-set attacks is important as more intelligent agents (e.g. spam filters and robots) are equipped with learning capability and can potentially be hacked via data they receive from the environment. This paper identifies the optimal training-set attack on a broad family of machine learners. First we show that optimal training-set attack can be formulated as a bilevel optimization problem. Then we show that for machine learners with certain Karush-Kuhn-Tucker conditions we can solve the bilevel problem efficiently using gradient methods on an implicit function. As examples, we demonstrate optimal training-set attacks on Support VectorMachines, logistic regression, and linear regression with extensive experiments. Finally, we discuss potential defenses against such attacks.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Towards the Science of Security and Privacy in Machine Learning (2016, background): “…introduce a more general framework for poisoning, which ﬁnds the optimal changes to the training set in terms of cardinality or the Frobenius norm, as long as the targeted ML model is trained using a convex loss (e.g., linear and logistic regression or SVMs) and its input domain is continuous [47].”
Understanding Black-box Predictions via Influence Functions (2017, methodology): “This attack is mathematically equivalent to the gradient-based training set attacks explored by Biggio et al. (2012); Mei & Zhu (2015b) and others in the context of different models.”
Advances and Open Problems in Federated Learning (2019, positive): “statements addressing this. One possible approach would be to use insights and techniques from work on machine teaching (see [438] for reference) to understand “optimal” data poisoning attacks, as in [292]. Any formal statement will likely depend on quantities such as the number of corrupted clients”
Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “A common trick used to solve the given bilevel optimization problems is to replace the inner optimization by its equilibrium conditions [24, 27, 29, 31].”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “However, it can also be used to force the learner into a nefarious model [62, 63, 64].”
Can You Really Backdoor Federated Learning? (2019, background): “attacks (5; 28; 33; 25; 18), the attacker can change a subset of all the training samples which is unknown to the learner.”
Data Poisoning against Differentially-Private Learners: Attacks and Defenses (2019, background): “[Biggio et al., 2012; Mei and Zhu, 2015; Xiao et al., 2015]) because the differentially-private learner produces a stochastic model, so the objective takes the expected value of C.”
Analyzing Federated Learning through an Adversarial Lens (2018, background): “We focus on an adversary which directly performs model poisoning instead of data poisoning (Biggio et al., 2012; Rubinstein et al., 2009; Mei & Zhu, 2015; Xiao et al., 2015; Mei & Zhu, 2015; Koh & Liang, 2017; Chen et al., 2017a; Jagielski et al., 2018) as the agents’ data is never shared with the…”
Stronger data poisoning attacks break data sanitization defenses (2018, methodology): “(2012b), and has subsequently been extended to linear and logistic regression (Mei and Zhu, 2015b), topic modeling (Mei & Zhu, 2015a), collaborative filtering (Li et al., 2016), and neural networks (Koh & Liang, 2017; Yang et al., 2017; MuñozGonzález et al.”
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (2017, background): “A general algorithmic framework for such poisoning attack instance optimization on traditional machine learning models is summarized in [45].”
Certified Defenses for Data Poisoning Attacks (2017, methodology): “Separately, the bound L (ˆ θ ) (cid:47) M was useful because M admits the natural minimax formulation (5), but the worst-case L (ˆ θ ) can be expressed directly as a bilevel optimization problem (Mei and Zhu, 2015b), which is intractable in general but admits a number of heuristics (Bard, 1999).”
Machine Teaching: An Inverse Problem to Machine Learning and an Approach Toward Optimal Education (2015, background): “As mentioned earlier, machine teaching also describes the optimal attack strategy if a hacker wants to inﬂuence a learning agent, see (Mei and Zhu 2015) and the references therein.”
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning (2018, methodology): “ence of higher poisoning rates would be an ill-posed task, if not infeasible at all [5], [26], [37], [39], [54], [55].”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, methodology): “ different learning algorithms. More systematic attacks, based on the exploitation of KKT conditions to solve the bilevel problem corresponding to poisoning attacks have been subsequently proposed in [8, 20, 23, 39]. In particular, Biggio et al. [8] have been the first to demonstrate the vulnerabili”
Data Poisoning Attacks on Factorization-Based Collaborative Filtering (2016, methodology): “Similar techniques were later generalized to optimize data poisoning attacks for several other important learning algorithms, such as Lasso regression [7], topic modeling [8], and autoregressive models [17].”
Adversarial Detection of Flash Malware: Limitations and Open Issues (2017, background): “In the remainder of this work we focus on classiﬁer evasion , while we refer the reader to [11], [22], [23], [35] for further details on classiﬁer poisoning .”
Stable and Fair Classification (2019, background): “Stable classiﬁcation algorithms can also provide defense for data poisoning attacks, whereby adversaries want to corrupt the learned model by injecting false training data [9, 53, 67].”