When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks

By Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daumé Iii, Tudor Dumitras. 2018

323 citations recorded by the source corpus.

Attacks against machine learning systems represent a growing threat as highlighted by the abundance of attacks proposed lately. However, attacks often make unrealistic assumptions about the knowledge and capabilities of adversaries. To evaluate this threat systematically, we propose the FAIL attacker model, which describes the adversary's knowledge and control along four dimensions. The FAIL model allows us to consider a wide range of weaker adversaries that have limited control and incomplete knowledge of the features, learning algorithms and training instances utilized. Within this framework, we evaluate the generalized transferability of a known evasion attack and we design StingRay, a targeted poisoning attack that is broadly applicable---it is practical against 4 machine learning applications, which use 3 different learning algorithms, and it can bypass 2 existing defenses. Our evaluation provides deeper insights into the transferability of poison and evasion samples across models and suggests promising directions for investigating defenses against this threat.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, methodology): “Our threat model is inspired by multiple existing studies [30, 47, 48, 56] on adversarial machine learning.”
Stronger data poisoning attacks break data sanitization defenses (2018, background): “In high-stakes settings like autonomous driving (Gu et al., 2017), biometrics (Chen et al., 2017), and cybersecurity (Rubinstein et al., 2009; Suciu et al., 2018), machine learning (ML) systems need to be secure against attacks by malicious actors.”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, neutral): “, [4, 33, 56]).”
Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks (2018, background): “The most closely related work to our own is by Suciu et al. [2018], who studies targeted attacks on neural nets.”
ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models (2018, background): “Besides the above, many recent works have tackled security and privacy in machine learning from various perspectives [39], [16], [9], [37], [7], [27], [20], [45], [36], [21], [40], [29], [41], [20].”
Transferable Clean-Label Poisoning Attacks on Deep Neural Nets (2019, methodology): “In general, data poisoning attacks aim to control the model’s behavior during inference by modifying its training data (Shafahi et al., 2018; Suciu et al., 2018; Koh & Liang, 2017; Mahloujifar et al., 2017).”
GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models (2023, background): “While the feature extractors may still be similar because of the well-known transferability property between large models [18, 58, 79, 84, 101], their differences could reduce the efficacy of cloaking.”
Data Poisoning Attacks Against Federated Learning Systems (2020, methodology): “Several poisoning attacks were developed for popular ML models including SVM [6,12,44,45,50,52], regression [19], dimensionality reduction [51], linear classifiers [12,23,57], unsupervised learning [7], and more recently, neural networks [12,30,42,45,53,58].”
Dynamic Backdoor Attacks Against Machine Learning Models (2020, background): “Attacks Against Machine Learning: Poisoning attack [4, 19, 40, 49] is another training time attack, in which the adversary manipulates the training data to compromise the target model.”
PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learning (2022, background): “In-processing defenses: Some in-processing defenses [5, 51, 52] train an initial model using the potentially poisoned training data, remove potentially poisoned training examples based on the initial model, and re-train a model using the remaining training examples.”
Fawkes: Protecting Privacy against Unauthorized Deep Learning Models (2020, background): “fectiveness of cloaking against any tracker models relies on the transferability effect, the property that models trained for similar tasks share similar properties and vulnerabilities, even when they were trained on different architectures and different training data [13, 38, 49, 69].”
Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples (2019, background): “Evasion attacks have been demonstrated to be highly successful for a number of classifiers [13, 18, 40, 45, 73, 88, 97, 110, 113, 114, 118].”
FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping (2020, unknown): “[17], [25], [45], neural networks [11], [18], [27], [30], [36], [37], and graph-based methods [20], [41], [49], as well as distributed privacy-preserving data analytics [10], [14].”
Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorization (2021, background): “These two types of attacks directly threaten the privacy of model users, while there are also many active attacks (Suciu et al., 2018; Yao et al., 2019) that lead DL models to produce abnormal behaviors.”
Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks (2018, background): “Poisoning integrity attacks have the goal of modifying predictions on a few targeted points by manipulating the training process [20,41].”
Exploring Backdoor Poisoning Attacks Against Malware Classifiers (2020, methodology): “[47] proposed a framework for defining attacker models in the poisoning space, and developed StingRay, a multi-model target poisoning attack methodology.”
Hidden Trigger Backdoor Attacks (2019, background): “(Suciu et al. 2018; Zhu et al. 2019) studied generalization and transferability of the poisoning attacks.”
A Comprehensive Survey on Poisoning Attacks and Countermeasures in Machine Learning (2022, methodology): “In targeted poisoning attacks, the adversarial goal is to force the target model to produce abnormal predictions on particular inputs [74, 111, 119].”