ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models
By Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, Cho-Jui Hsieh. 2017
2,214 citations recorded by the source corpus.
Deep neural networks (DNNs) are one of the most prominent technologies of our time, as they achieve state-of-the-art performance in many machine learning tasks, including but not limited to image classification, text mining, and speech processing. However, recent research on DNNs has indicated ever-increasing concern on the robustness to adversarial examples, especially for security-critical tasks such as traffic sign identification for autonomous driving. Studies have unveiled the vulnerability of a well-trained DNN by demonstrating the ability of generating barely noticeable (to both human and machines) adversarial images that lead to misclassification. Furthermore, researchers have shown that these adversarial images are highly transferable by simply training and attacking a substitute model built upon the target model, known as a black-box attack to DNNs. Similar to the setting of training substitute models, in this paper we propose an effective black-box attack that also only has access to the input (images) and the output (confidence scores) of a targeted DNN. However, different from leveraging attack transferability from substitute models, we propose zeroth order optimization (ZOO) based attacks to directly estimate the gradients of the targeted DNN for generating adversarial examples. We use zeroth order stochastic coordinate descent along with dimension reduction, hierarchical attack and importance sampling techniques to efficiently attack black-box models. By exploiting zeroth order optimization, improved attacks to the targeted DNN can be accomplished, sparing the need for training substitute models and avoiding the loss in attack transferability. Experimental results on MNIST, CIFAR10 and ImageNet show that the proposed ZOO attack is as effective as the state-of-the-art white-box attack (e.g., Carlini and Wagner's attack) and significantly outperforms existing black-box attacks via substitute models.
Subfields: Artificial Intelligence, Molecular Biology, Virology, Safety, Robustness & Interpretability, Mathematical Optimization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Bernt Schiele
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Nicholas Carlini
- Mohit Bansal
- Jun Zhu
- Matthias Hein
In Molecular Biology
- Matthias Hein
- Tianyu Pang
- Jun Zhu
- Yinpeng Dong
- Hang Su
- Francesco Croce
- Nicholas Carlini
- Cho-Jui Hsieh
In Virology
- Jianbo Chen
- Michael I. Jordan
- Cho-Jui Hsieh
- Pin-Yu Chen
- Huan Zhang
- Jun Zhu
- Aleksander Madry
- Tianyu Pang
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Matthias Hein
- Tianyu Pang
- Jun Zhu
- Cho-Jui Hsieh
- Florian Tramèr
- Nicolas Papernot
- Aleksander Madry
In Mathematical Optimization
- Huan Zhang
- Cho-Jui Hsieh
- Michael I. Jordan
- Jianbo Chen
- Pin-Yu Chen
- Jun Zhu
- Aleksander Madry
- Jinfeng Yi
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, background): “zero accuracy on image classification benchmarks such as CIFAR-10 or ImageNet [90]. In the black-box setting, models have also been shown to be vulnerable to attacks based on query-access to the model [99, 82] or based on substitute models trained on similar data [377, 315, 386]. While black-box atta”
- Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, background): “This includes black-box variants of JSMA (Narodytska & Kasiviswanathan, 2016) and of the Carlini & Wagner attack (Chen et al., 2017) as well as generator networks that predict adversarials (Hayes & Danezis, 2017).”
- Ensemble Adversarial Training: Attacks and Defenses (2017, methodology): “If queries to the target model yield prediction confidences , an adversary can estimate the target’s gradient at a given point (e.g., using finite-differences as in Chen et al. (2017)) and fool the target with our R+FGSM attack.”
- Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness Against Adversarial Attack (2018, methodology): “For ZOO attack[11], we test our defense on 200 randomly selected test samples for an untargeted attack.”
- Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach (2018, methodology): “Adversarial examples can also be crafted from a black-box network model using an ensemble approach (Liu et al., 2016), training a substitute model (Papernot et al., 2017), or employing zeroth-order optimization based attacks (Chen et al., 2017c).”
- Adversarial Detection of Flash Malware: Limitations and Open Issues (2017, background): “ the presence of obfuscated malware. We then evaluate FlashBuster robustness by simulating evasion attacks that leverage the knowledge that the attacker may possess about the targeted learning system [9, 8, 27, 19, 28, 29, 30], against an increasing number of modications to the input samples. The co”
- Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “[41] also proposed ‘Zeroth Order Optimization (ZOO)’ based attacks that directly estimate the gradients of the targeted model for generating the adversarial examples.”
- Adversarial Risk and the Dangers of Evaluating Against Weak Attacks (2018, methodology): “s v−1 to have finite first and second moments [Spall, 1992]. Ininitial experiments, we alsotried natural evolutionary strategies [Wierstra et al., 2008, Ilyas et al., 2017] and zero-order optimization [Chen et al., 2017], two other gradient-free optimization techniques. Comparisons are included in the”
- EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples (2017, methodology): “In (Chen et al. 2017), an effective black-box C&W attack is made possible using zeroth order optimization (ZOO).”
- Technical Report on the CleverHans v2.1.0 Adversarial Examples Library (2016, methodology): “Gradients are approximated using finite difference estimates [8] in random directions.”
- Disentangling Adversarial Robustness and Generalization (2018, background): “), to black-box attacks [17, 10, 96, 38, 80, 67], with limited access to model queries.”
- Towards Fast Computation of Certified Robustness for ReLU Networks (2018, background): “The existence of adversarial examples poses a huge threat to the application of DNNs in mission-critical tasks including security cameras, self-driving cars and aircraft control systems.”
- Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (2018, methodology): “e context of image classification. Since then, a number of approacheshave surfaced regardingadversarialattacks on CNNs, such as using gradient-basedmethods [2, 11, 12, 13, 14, 15], score-based methods [16, 17, 18], transfer-based methods [19], and decision-based methods [20]. In addition to image cla”