Black-box Adversarial Attacks with Limited Queries and Information

By Andrew Ilyas, Logan Engstrom, Anish Athalye, Jessy Lin. 2018

1,400 citations recorded by the source corpus.

Current neural network-based classifiers are susceptible to adversarial examples even in the black-box setting, where the attacker only has query access to the model. In practice, the threat model for real-world systems is often more restrictive than the typical black-box model where the adversary can observe the full output of the network on arbitrarily many chosen inputs. We define three realistic threat models that more accurately characterize many real-world classifiers: the query-limited setting, the partial-information setting, and the label-only setting. We develop new attacks that fool classifiers under these more restrictive threat models, where previous methods would be impractical or ineffective. We demonstrate that our methods are effective against an ImageNet classifier under our proposed threat models. We also demonstrate a targeted black-box attack against a commercial classifier, overcoming the challenges of limited query access, partial information, and other practical issues to break the Google Cloud Vision API.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Adversarial Example Games (2020, methodology): “…estimation is an attractive approach with notable approaches utilizing black box optimization schemes such as Finite Differences [15], Natural Evolutionary Strategies [39, 42], learned priors in a bandit optimization framework [40], meta-learning attack patterns [23], and query efﬁcient Defenses .”
Towards the first adversarially robust neural network model on MNIST (2018, methodology): “We use a simple coordinate-wise finite difference method (NES estimates [Ilyas et al., 2018] performed comparable or worse) and repeat the attacks with different values for the step size of the gradient estimator.”
On Evaluating Adversarial Robustness (2019, background): “• NES (Ilyas et al., 2018a) is effective at generating adversarial examples with a limited number of queries, and so generates adversarial examples of higher distortion.”
Disentangling Adversarial Robustness and Generalization (2018, neutral): “These adversarial attacks pose severe security threats, as demonstrated against Clarifai.com [56, 8] or Google Cloud Vision [36].”
Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks (2018, background): “We finally remark that non-differentiable learning algorithms, like decision trees and random forests, can be attacked with more complex strategies [17,19] or using gradient-based optimization against a differentiable surrogate learner [31,37].”
Adversarial Robustness vs. Model Compression, or Both? (2019, background): “, structure and weight parameters) [7, 8, 50, 44]; or black-box accesses where the adversary can only make queries and observe outputs [9, 22].”
Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning (2020, background): “There is a rapidly growing literature on test-time attacks on neural-network image classifiers [3, 37, 38, 49, 69]; some employ only black-box model queries [15, 33] rather than gradient-based optimization.”
Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection (2019, methodology): “Here, we kept the settings the same as those described in the original work [13] and changed the optimization objective to seek a trigger that can significantly lower the test statistic J∗ of the target class.”
HopSkipJumpAttack: A Query-Efficient Decision-Based Attack (2019, methodology): “Chen et al. [8] and Ilyas et al. [9, 10] introduced score-based methods using zeroth-order gradient estimation to craft adversarial examples.”
Overcoming Negative Transfer: A Survey (2020, background): “Adversarial examples, which are perturbed inputs that aim to mislead DNNs to make mistakes, usually with high adversarial transferability, is an intersecting phenomena in current deep learning literature [100].”
Convergence of Adversarial Training in Overparametrized Neural Networks (2019, neutral): “For instance, [13, 24, 8] considered the black-box setting where the model is hidden but the attacker can make queries and get the corresponding outputs of the model.”
Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets (2020, methodology): “Gradient estimation methods estimate the gradients of the target model using black-box optimization methods such as Finite Differences (FD) (Chen et al., 2017; Bhagoji et al., 2018) or Natural Evolution Strategies (NES) (Ilyas et al., 2018; Jiang et al., 2019).”
Black-Box Adversarial Attack with Transferable Model-based Embedding (2019, methodology): “As in (Ilyas et al., 2018a; Wierstra et al., 2014), we can employ NES to approximate the gradient of a properly deﬁned surrogate loss in order to ﬁnd an adversarial example.”
Bag of Tricks for Adversarial Training (2020, background): “These include transfer-based attacks (Dong et al., 2018; 2019; Cheng et al., 2019b), quasi-gradient attacks (Chen et al., 2017a; Uesato et al., 2018; Ilyas et al., 2018), and decision-based attacks (Brendel et al., 2018; Cheng et al., 2019a).”
Adversarial Examples on Object Recognition (2020, background): “Other publications use genetic algorithms to synthesize adversarial examples [3] or to estimate the gradients [85] in order to generate adversarial examples in limited domains (such as limited queries or information).”
On Adaptive Attacks to Adversarial Example Defenses (2020, methodology): “Score-based attacks (Ilyas et al., 2018; Chen et al., 2017) might provide better gradients than BPDA, if combined with EOT.”
Towards Deep Learning Models Resistant to Large Perturbations (2020, background): “Subsequently, these defenses are easily broken by alternative adversaries, such as non-gradient based (Chen et al., 2017; Ilyas et al., 2018; Uesato et al., 2018), or blackbox attacks (Papernot et al.”