One Pixel Attack for Fooling Deep Neural Networks

By Jiawei Su, Danilo Vasconcellos Vargas, Kouichi Sakurai. 2017

2,581 citations recorded by the source corpus.

Recent research has revealed that the output of deep neural networks (DNNs) can be easily altered by adding relatively small perturbations to the input vector. In this paper, we analyze an attack in an extremely limited scenario where only one pixel can be modified. For that we propose a novel method for generating one-pixel adversarial perturbations based on differential evolution (DE). It requires less adversarial information (a black-box attack) and can fool more types of networks due to the inherent features of DE. The results show that 67.97% of the natural images in Kaggle CIFAR-10 test dataset and 16.04% of the ImageNet (ILSVRC 2012) test images can be perturbed to at least one target class by modifying just one pixel with 74.03% and 22.91% confidence on average. We also show the same vulnerability on the original CIFAR-10 dataset. Thus, the proposed attack explores a different take on adversarial machine learning in an extreme limited scenario, showing that current DNNs are also vulnerable to such low dimension attacks. Besides, we also illustrate an important application of DE (or broadly speaking, evolutionary computation) in the domain of adversarial machine learning: creating tools that can effectively generate low-cost adversarial attacks against neural networks for evaluating robustness.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Making Convolutional Networks Shift-Invariant Again (2019, background): “These perturbations can be on pixels (Goodfellow et al., 2014a;b), a single pixel (Su et al., 2019), small deformations (Xiao et al., 2018), or even afﬁne transformations (Engstrom et al., 2017).”
Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, background): “, by synthesizing an entire image [29], a small patch [18, 13], a few pixels [8], or only a single pixel [40].”
Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “This is closely analogous to the reason that machine learning models can fail to generalize or be susceptible to adversarial examples[19]—there are many more ways of classifying data that do well in training than any specific way the programmers had in mind.”
Why do deep convolutional networks generalize so poorly to small image transformations? (2018, background): “Despite the excellent performance of CNNs on object recognition, the vulnerability to adversarial attacks suggests that superficial changes can result in highly non-human shifts in prediction [43, 35, 26, 1, 15, 41, 3, 40].”
STRIP: a defence against trojan attacks on deep neural networks (2019, background): “have full control over converting the physical scene into an effective adversarial digital input; perturbations in the digital input is small, for example, the one-pixel adversarial example attack in [14]. Thus, a camera will not be able to perceive such perturbations due to sensor imperfections [13”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “Currently, there are already some works of attack on Convolutional Neural Network (CNN) [72]–[74], Recurrent Neural Networks (RNN) [75] and Deep Reinforcement Learning (DeepRL) [76], [77] and GANs [78].”
What do we need to build explainable AI systems for the medical domain? (2017, background): “”Potentially interpretable” could also mean ”accurate”, see for example the recent work ”One pixel attack for fool-ing deep neural networks” by Su et al. (2017).”
Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks (2018, background): “Moreover, for more than half of the test data, we could ensure that there are no one-pixel attacks [28].”
Ensemble Bayesian Decision Making with Redundant Deep Perceptual Control Policies (2018, background): “Even small perturbations to the input can alter the output of Deep NNs [7], [8] and the Deep NNs are easily fooled [9].”
Adversarially Robust Generalization Requires More Data (2018, background): “However, changes in different `p norms [11, 36, 49], sparse perturbations [10, 37, 39, 48], or mild spatial transformations can also be imperceptible to a human [55].”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “[68] claimed successful fooling of three different network models on 70.”
Adversarial Patch (2017, background): “[6] recently demonstrated that modifying 1 pixel on a 32x32 pixel CIFAR-10 image (0.”
Are adversarial examples inevitable? (2018, background): “A number of papers have looked at sparse adversarial examples, in which a small number of image pixels, in some cases only one (Su et al., 2017), are changed to manipulate the class label.”
Disentangling Adversarial Robustness and Generalization (2018, background): “), to black-box attacks [17, 10, 96, 38, 80, 67], with limited access to model queries.”