Delving into Transferable Adversarial Examples and Black-box Attacks

By Yan-Po Liu, Xinyun Chen, Chang Liu, Dawn Song. 2016

1,962 citations recorded by the source corpus.

An intriguing property of deep neural networks is the existence of adversarial examples, which can transfer among different architectures. These transferable adversarial examples may severely hinder deep neural network-based applications. Previous works mostly study the transferability using small scale datasets. In this work, we are the first to conduct an extensive study of the transferability over large models and a large scale dataset, and we are also the first to study the transferability of targeted adversarial examples with their target labels. We study both non-targeted and targeted adversarial examples, and show that while transferable non-targeted adversarial examples are easy to find, targeted adversarial examples generated using existing approaches almost never transfer with their target labels. Therefore, we propose novel ensemble-based approaches to generating transferable adversarial examples. Using such approaches, we observe a large proportion of targeted adversarial examples that are able to transfer with their target labels for the first time. We also present some geometric studies to help understanding the transferable adversarial examples. Finally, we show that the adversarial examples generated using ensemble-based approaches can successfully attack this http URL, which is a black-box image classification system.

Subfields: Artificial Intelligence, Hardware and Architecture, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Safety, Robustness & Interpretability

Citations in the corpus

Red Teaming Language Models with Language Models (2022, methodology): “Adversarial Transfer: Adversarial inputs often transfer across models (Szegedy et al., 2014; Liu et al., 2017; Perez et al., 2019), in which case it is easy for adversaries to attack a new model if they have attacked others.”
Adversarial Example Games (2020, background): “Indeed, this setting—as well as the general notion of transferring attacks between classiﬁers—has generated an increasing amount of empirical interest [22, 49, 70, 68], though the ﬁeld currently lacks the necessary theoretical foundations to understand the feasibility of such attacks.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Adversarial examples can be generated using access to model parameters, also known as white-box attacks, or without such access, with black-box attacks (Papernot et al., 2016a, 2017; Narodytska and Kasiviswanathan, 2017; Liu et al., 2017).”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, methodology): “…important distinction is often drawn between white-box attacks, where adversarial examples are generated with access to the model parameters, and black-box attacks, where examples are generated without such access (Papernot et al., 2016; 2017; Liu et al., 2017; Narodytska & Kasiviswanathan, 2017).”
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “These can be applied to DNNs during inference [38], [39], [40], [41], [42].”
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning (2017, background): “In particular, a body of literature studying adversarial examples belongs to this category [29], [53], [36], [48], [54], [52], [15], [40], [25].”
Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, background): “If adversarial examples are created on an ensemble of substitute models the success rate on the attacked model can reach up to 100% in certain scenarios (Liu et al., 2016).”
Attacking the Madry Defense Model with L1-based Adversarial Examples (2017, background): “We expect that if an adversarial example remains adversarial to multiple models, it is more likely to transfer to other unseen models (Liu et al., 2016; Papernot et al., 2016).”
ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, methodology): “For instance, the authors in [26] have demonstrated a successful black-box adversarial attack to Clarifai.”
Ensemble Adversarial Training: Attacks and Defenses (2017, background): “[19] showed that these inputs transfer (or generalize) across models, thus enabling black-box attacks on deployed models [13, 9]: an adversary can train a local model (possibly by issuing prediction queries to the target model [13, 21]), find an adversarial example on the local model, and apply it t”
Backdooring Convolutional Neural Networks via Targeted Weight Perturbations (2018, background): “The other, more prevalent class of attacks leveraging perturbations that has been studied is referred to as adversarial examples [10, 19, 7, 31, 21, 23, 25, 29].”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, background): “However, existing studies [6, 7, 12, 19, 20, 22, 28, 31, 32, 34, 39, 42, 45, 48, 51, 53– 55, 58, 62, 63, 70, 71, 73] on adversarial machine learning mainly focused on machine learning for non-graph data, with only a few studies [8, 15–17, 41, 57, 61, 77–79] as exceptions.”
When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks (2018, background): “Some of the recent evasion attacks [28, 36] investigate the transferability property of their solutions.”
Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification (2017, background): “A number of recent studies [2, 3, 11, 15, 17, 18, 22] have demonstrated that DNNs are vulnerable to evasion attacks at the testing phase.”
Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness Against Adversarial Attack (2018, methodology): “substitute [34], ZOO [11] and transferable [29] attack) in a Section 4.”
Formal Security Analysis of Neural Networks using Symbolic Intervals (2018, methodology): “Existing methods to test DNNs against corner cases focus on finding adversarial examples [7, 11, 27, 28, 33, 35, 37, 38, 44] without providing formal guarantees about the non-existence of adversarial inputs even within very small input ranges.”
Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models (2018, background): “In the context of classiﬁcation, these perturbations cause the legitimate sample to be misclassiﬁed at inference time (Szegedy et al., 2014; Goodfellow et al., 2015; Papernot et al., 2016b; Liu et al., 2017).”