Explicit Defense Actions Against Test-Set Attacks

By Scott Alfeld, Xiaojin Zhu, Paul R. Barford. 2017

52 citations recorded by the source corpus.

Automated learning and decision making systems in public-facing applications are vulnerable to malicious attacks. Examples of such systems include spam detectors, credit card fraud detectors, and network intrusion detection systems. These systems are at further risk of attack when money is directly involved, such as market forecasters or decision systems used in determining insurance or loan rates. In this paper, we consider the setting where a predictor Bob has a fixed model, and an unknown attacker Alice aims to perturb (or poison) future test instances so as to alter Bob's prediction to her benefit. We focus specifically on Bob's optimal defense actions to limit Alice's effectiveness. We define a general framework for determining Bob's optimal defense action against Alice's worst-case attack. We then demonstrate our framework by considering linear predictors, where we provide tractable methods of determining the optimal defense action. Using these methods, we perform an empirical investigation of optimal defense actions for a particular class of linear models -- autoregressive forecasters -- and find that for ten real world futures markets, the optimal defense action reduces the Bob's loss by between 78 and 97%.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Data Poisoning against Differentially-Private Learners: Attacks and Defenses (2019, positive): “ quantitative bounds on how much an adversary can change the distribution over learned models by manipulating a ﬁxed number of training items. We are not the ﬁrst to design defenses to data poisoning [19, 18, 2], but our proposal has several notable strengths compared to previous work. First, our de”
Teacher Improves Learning by Selecting a Training Subset (2018, background): “Machine teaching applications include education [14, 33, 39, 27, 13, 34], computer security [2, 1, 32], and interactive machine learning [40, 12, 24].”
An Overview of Machine Teaching (2018, background): “• In security applications, this means the victim may employ defense mechanisms to resist attacks [3].”
Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications (2018, unknown): “Future work includes applying and extending our proposed machine teaching framework to quantify the inherent difficulty of teaching common IRL benchmark tasks, to improve how robots convey their objectives to humans [14], to help robots better learn from noisy but informative human demonstrations, a”