Simple Black-Box Adversarial Perturbations for Deep Networks

By Nina Narodytska, Shiva Prasad Kasiviswanathan. 2016

247 citations recorded by the source corpus.

Deep neural networks are powerful and popular learning models that achieve state-of-the-art pattern recognition performance on many computer vision, speech, and language processing tasks. However, these networks have also been shown susceptible to carefully crafted adversarial perturbations which force misclassification of the inputs. Adversarial examples enable adversaries to subvert the expected system behavior leading to undesired consequences and could pose a security risk when these systems are deployed in the real world. In this work, we focus on deep convolutional neural networks and demonstrate that adversaries can easily craft adversarial examples even without any internal knowledge of the target network. Our attacks treat the network as an oracle (black-box) and only assume that the output of the network can be observed on the probed inputs. Our first attack is based on a simple idea of adding perturbation to a randomly selected single pixel or a small set of them. We then improve the effectiveness of this attack by carefully constructing a small set of pixels to perturb by using the idea of greedy local-search. Our proposed attacks also naturally extend to a stronger notion of misclassification. Our extensive experimental results illustrate that even these elementary attacks can reveal a deep neural network's vulnerabilities. The simplicity and effectiveness of our proposed schemes mean that they could serve as a litmus test while designing robust networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Adversarial Examples for Evaluating Reading Comprehension Systems (2017, background): “In contrast with prior work in computer vision (Papernot et al., 2017; Narodytska and Kasiviswanathan, 2016; Moosavi-Dezfooli et al., 2017), A DD O NE S ENT does not require any access to the model or to any training data: it generates adversarial examples based solely on the intuition that…”
Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, background): “This includes black-box variants of JSMA (Narodytska & Kasiviswanathan, 2016) and of the Carlini & Wagner attack (Chen et al., 2017) as well as generator networks that predict adversarials (Hayes & Danezis, 2017).”
Adversarially Robust Generalization Requires More Data (2018, background): “However, changes in different `p norms [11, 36, 49], sparse perturbations [10, 37, 39, 48], or mild spatial transformations can also be imperceptible to a human [55].”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “Narodytska andKasiviswanathan [54] also introduced further black-box attacks that have been found effective in fooling the neural networks by changing only few pixel values in the images.”
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (2018, background): “Since then, a number of approaches have surfaced regarding adversarial attacks on CNNs, such as using gradient-based methods [2, 11, 12, 13, 14, 15], score-based methods [16, 17, 18], transfer-based methods [19], and decision-based methods [20].”
Adversarial Image Perturbation for Privacy Protection A Game Theory Perspective (2017, background): “Many AIP techniques assume a full knowledge on the model f , but the computation of blackbox AIPs is another active research field [29, 30, 24, 16]; U can potentially adopt these methods.”
advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch (2019, unknown): “Besides gradient-based attacks, the current version of advertorch also implements • SinglePixelAttack, LocalSearchAttack (Narodytska and Kasiviswanathan, 2016), and • JacobianSaliencyMapAttack (Papernot et al.”
Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Models (2018, background): “Previous works have distinguished between modelaware strategies, where the adversarial algorithms have access to the model parameters, and modelagnostic strategies, where the adversary does not have such information (Papernot et al., 2017; Liu et al., 2017; Narodytska and Kasiviswanathan, 2017).”
Feature-Guided Black-Box Safety Testing of Deep Neural Networks (2017, background): “[27] randomly searches the vector space around the input image for changes which will cause a misclassification.”
Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey (2023, background): “Narodytska and Kasiviswanathan [99] apply local-search algorithms [101] to measure the inﬂuence of a subset of pixels on the output and implicitly compute an approximation to the gradient.”
AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networks (2018, background): “Data-drivenacceleration [34] local random search score [20] substitute model score [35] acquisition via posterior score [23] Gaussian perturbation decision [11] natural evolution strategy score [3] coordinate-wisegradientestimation score [4] coordinate-wisegradientestimation score AutoZOOM (this…”
A Geometric Perspective on the Transferability of Adversarial Directions (2018, background): “As demonstrated by Moosavi-Dezfooli et al. (2017a), Narodytska and Kasiviswanathan (2016), and Papernot et al. (2016), adversarial examples often transfer to other classiﬁers.”
Black-box Adversarial Attacks with Limited Queries and Information (2018, background): “Narodytska & Kasiviswanathan (2017) propose a black-box gradient estimation attack using a local-search based technique, showing that perturbing only a small fraction of pixels in an image is often sufﬁcient for it to be misclassi-ﬁed.”
Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples (2019, background): “Black-box with query access: This is a more restrictive threat model, where the adversary is only assumed to have access to the output probability distribution д(·) for any input x ∈ X [12, 25, 31, 60, 93].”