Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples

By Nicolas Papernot, Patrick Drew McDaniel, Ian J. Goodfellow. 2016

1,918 citations recorded by the source corpus.

Many machine learning models are vulnerable to adversarial examples: inputs that are specially crafted to cause a machine learning model to produce an incorrect output. Adversarial examples that affect one model often affect another model, even if the two models have different architectures or were trained on different training sets, so long as both models were trained to perform the same task. An attacker may therefore train their own substitute model, craft adversarial examples against the substitute, and transfer them to a victim model, with very little information about the victim. Recent work has further developed a technique that uses the victim model as an oracle to label a synthetic training set for the substitute, so the attacker need not even collect a training set to mount the attack. We extend these recent techniques using reservoir sampling to greatly enhance the efficiency of the training procedure for the substitute model. We introduce new transferability attacks between previously unexplored (substitute, victim) pairs of machine learning model classes, most notably SVMs and decision trees. We demonstrate our attacks on two commercial machine learning classification systems from Amazon (96.19% misclassification rate) and Google (88.94%) using only 800 queries of the victim model, thereby showing that existing machine learning approaches are in general vulnerable to systematic black-box attacks regardless of their structure.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Adversarial Examples Are Not Bugs, They Are Features (2019, background): “One of the most intriguing properties of adversarial examples is that they transfer across models with different architectures and independently sampled training sets [Sze+14; PMG16; CRP19].”
Red Teaming Language Models with Language Models (2022, methodology): “…it is possible to create adversarial examples for a target model by creating adversarial examples using another model (Szegedy et al., 2014; Liu et al., 2017; Perez et al., 2019), especially when the other model is trained to make similar predictions as the target model (Papernot et al., 2016a,b).”
Online Adversarial Attacks (2021, background): “(2017), where the attacker has complete access, and the less permissive blackbox threat model where an attacker only has partial information, like the ability to query the target model (Chen et al., 2017; Ilyas et al., 2019; Papernot et al., 2016).”
Adversarial Example Games (2020, methodology): “This setting is more permissive than the semi-whtiebox and the blackbox setting, the latter of which the attacker has only access to the prediction [54, 55] or sometimes the predicted conﬁdence [33].”
Towards the Science of Security and Privacy in Machine Learning (2016, background): “In a follow-up work [61], they show that the attack generalizes to many ML models by having a logistic regression oracle trained by Amazon misclassify 96% of the adversarial examples crafted.”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, background): “This is a sequence-to-sequence model with attention (Sutskever et al., 2014; Bahdanau et al., 2014) that is trained on characters to characters ( char2char ).”
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, neutral): “C R ] 1 4 Se p 20 17”
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (2017, background): “In particular, a body of literature studying adversarial examples belongs to this category [29], [53], [36], [48], [54], [52], [15], [40], [25].”
Attacking the Madry Defense Model with L1-based Adversarial Examples (2017, background): “We expect that if an adversarial example remains adversarial to multiple models, it is more likely to transfer to other unseen models (Liu et al., 2016; Papernot et al., 2016).”
ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, methodology): “It is worth noting that under this black-box setting, existing attacking approaches tend to make use of the power of free query to train a substitute model [17, 34, 35], which is a representative substitute of the targeted DNN.”
Certified Defenses for Data Poisoning Attacks (2017, background): “…striking recent security vulnerability discovered in machine learning systems is adversarial test images that can fool image classiﬁers despite being imperceptible from normal images (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini et al., 2016; Kurakin et al., 2016; Papernot et al., 2016a).”
Adversarial Machine Learning at Scale (2016, background): “In particular, neural networks and many other categories of machine learning models are highly vulnerable to attacks based on small modiﬁcations of the input to the model at test time (Biggio et al., 2013; Szegedy et al., 2014; Goodfellow et al., 2014; Papernot et al., 2016b).”
Towards Evaluating the Robustness of Neural Networks (2016, background): “…the existence of adversarial examples not only limits their potential application settings, but entirely defeats its purpose: an adversary who is able to make only slight modifications to a malware file that cause it to remain malware, but become classified as benign, has entirely defeated the…”
When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks (2018, background): “For example, in [36] a black-box adversary possesses knowledge of the full feature representations, whereas its counterpart in [50] only assumes access to the raw data (i.”
On the (Statistical) Detection of Adversarial Examples (2017, methodology): “They compute a n − 1 dimensional hyperplane to separate the training points.”
Delving into Transferable Adversarial Examples and Black-box Attacks (2016, methodology): “Our work differs from Papernot et al. (2016a;b) (Stallkamp et al. (2012)); in our work, we study the transferability over larger models and a larger dataset, i.e., ImageNet.”