Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models

By Wieland Brendel, Jonas Rauber, Matthias Bethge. 2017

1,559 citations recorded by the source corpus.

Many machine learning algorithms are vulnerable to almost imperceptible perturbations of their inputs. So far it was unclear how much risk adversarial perturbations carry for the safety of real-world machine learning applications because most methods used to generate such perturbations rely either on detailed model information (gradient-based attacks) or on confidence scores such as class probabilities (score-based attacks), neither of which are available in most real-world scenarios. In many such cases one currently needs to retreat to transfer-based attacks which rely on cumbersome substitute models, need access to the training data and can be defended against. Here we emphasise the importance of attacks which solely rely on the final model decision. Such decision-based attacks are (1) applicable to real-world black-box models such as autonomous cars, (2) need less knowledge and are easier to apply than transfer-based attacks and (3) are more robust to simple defences than gradient- or score-based attacks. Previous attacks in this category were limited to simple models or simple datasets. Here we introduce the Boundary Attack, a decision-based attack that starts from a large adversarial perturbation and then seeks to reduce the perturbation while staying adversarial. The attack is conceptually simple, requires close to no hyperparameter tuning, does not rely on substitute models and is competitive with the best gradient-based attacks in standard computer vision tasks like ImageNet. We apply the attack on two black-box algorithms from Clarifai.com. The Boundary Attack in particular and the class of decision-based attacks in general open new avenues to study the robustness of machine learning models and raise new questions regarding the safety of deployed machine learning systems. An implementation of the attack is available as part of Foolbox at this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “We also run gradient-free black-box attacks; the boundary attack (Brendel et al., 2017) and the brute-force pointwise attack (Rauber et al.”
Towards Deep Learning Models Resistant to Adversarial Attacks (2017, methodology): “In the case of the MNIST adversarially trained networks, we also evaluate the performance of the Decision Boundary Attack (DBA) [4] with 2000 steps and PGD on standard and adversarially trained models.”
Unsolved Problems in ML Safety (2021, background): “Here, adversaries can feed in examples to an ML system and receive the system’s outputs, but they do not have access to the intermediate ML system computation [21].”
Adversarial Training and Robustness for Multiple Perturbations (2019, methodology): “(2) to detect gradient-masking, we use decision-based attacks: the Boundary Attack [3] for `2, the Pointwise Attack [31] for `1, and the Boundary Attack++ [7] for `∞; (3) for spatial attacks, we use the optimal attack of [11] that enumerates all small rotations and translations.”
Towards the first adversarially robust neural network model on MNIST (2018, methodology): “We demonstrate both quantitatively and qualitatively that—using the non-gradient-based Boundary Attack [Brendel et al., 2018]—we can find minimal L2 adversarial perturbations for Madry et al.”
Theoretically Principled Trade-off between Robustness and Accuracy (2019, background): “The boundary attack (Brendel et al., 2018) is a black-box attack method which searches for data points near the decision boundary and attack robust models by these data points.”
On Evaluating Adversarial Robustness (2019, background): “• The Boundary Attack (Brendel et al., 2017) is general-purpose hard-label attack and performs a descent along the decision boundary using a rejection sampling approach.”
Foolbox v0.8.0: A Python toolbox to benchmark the robustness of machine learning models (2017, background): “n the class decision of the model. They do not require gradients or probabilities. Boundary Attack foolbox.attacks.BoundaryAttack Foolbox provides the reference implementation for the BoundaryAttack (Brendel et al., 2018). The BoundaryAttack is the most effective decision-based adversarial attack to”
Unrestricted Adversarial Examples (2018, methodology): “We have selected three attacks for this warm-up, including two black-box pixel-wise optimization attacks [Uesato et al., 2018, Brendel et al., 2017] and a spatial attack [Engstrom et al., 2017].”
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (2018, background): “Since then, a number of approaches have surfaced regarding adversarial attacks on CNNs, such as using gradient-based methods [2, 11, 12, 13, 14, 15], score-based methods [16, 17, 18], transfer-based methods [19], and decision-based methods [20].”
Robust Perception through Analysis by Synthesis (2018, methodology): “There has been much work showing that basically all defenses suggested so far in the literature do not substantially increase robustness over vanilla networks [Carlini and Wagner, 2017, Brendel et al., 2018, Brendel and Bethge, 2017].”
Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey (2023, methodology): “In the original implementation of the Boundary Attack by Brendel et al. [100], the direction is drawn randomly from a proposal distribution.”
Foolbox: A Python toolbox to benchmark the robustness of machine learning models (2017, methodology): “BoundaryAttack Foolbox provides the reference implementation for the Boundary Attack (Brendel et al., 2018).”
The Pitfalls of Simplicity Bias in Neural Networks (2020, methodology): “Adversarial Defenses: Neural networks trained using standard procedures such as SGD are extremely vulnerable [22] to -bound adversarial attacks such as FGSM [22], PGD [40], CW [11], and Momentum [17]; Unrestricted attacks [7, 19] can significantly degrade model performance as well.”
Retrieval-Augmented Convolutional Neural Networks for Improved Robustness against Adversarial Examples (2018, methodology): “Similarly to CIFAR-10, the proposed approach is most robust to DeepFool and Bound-ary, while it is most susceptible to L-BFGS.”
Adversarial Attacks and Defences Competition (2018, methodology): “ to different image classiﬁers, we use multiple target models to train the attack FCN. We maximize the sum of losses of all models. In this competition, we used eight models: (1) ResNet50, (2) VGG16, (3) Inception v3, (4) Inception v3 with adversarial training, (5) Inception v3 with ensemble adversa”
Certified Adversarial Robustness with Additive Noise (2018, background): “A large body of work has been developed on defensive methods to tackle adversarial examples, yet most remain vulnerable to adaptive attacks [3, 4, 5, 6, 7, 8, 9, 10].”
Black-box Adversarial Attacks with Limited Queries and Information (2018, methodology): “Note that this is a generalization of the decision-only setting deﬁned in Brendel et al. (2018), where k = 1 , and the attacker only has access to the top label.”