Foolbox v0.8.0: A Python toolbox to benchmark the robustness of machine learning models

By Jonas Rauber, Wieland Brendel, Matthias Bethge. 2017

285 citations recorded by the source corpus.

Even todays most advanced machine learning models are easily fooled by almost imperceptible perturbations of their inputs. Foolbox is a new Python package to generate such adversarial perturbations and to quantify and compare the robustness of machine learning models. It is build around the idea that the most comparable robustness measure is the minimum perturbation needed to craft an adversarial example. To this end, Foolbox provides reference implementations of most published adversarial attack methods alongside some new ones, all of which perform internal hyperparameter tuning to find the minimum adversarial perturbation. Additionally, Foolbox interfaces with most popular deep learning frameworks such as PyTorch, Keras, TensorFlow, Theano and MXNet and allows different adversarial criteria such as targeted misclassification and top-k misclassification as well as different distance measures. The code is licensed under the MIT license and is openly available at this https URL . The most up-to-date documentation can be found at this http URL .

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (2019, background): “Its popularity has often led “adversarial robustness” to become interchangeable with “robustness” in the literature (Bastani et al., 2016; Rauber et al., 2017).”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, neutral): “We also run gradient-free black-box attacks; the boundary attack (Brendel et al., 2017) and the brute-force pointwise attack (Rauber et al., 2017).”
Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models (2017, methodology): “We use the implementation in Foolbox 0.10.0 (Rauber et al., 2017).”
Adversarial Vulnerability of Neural Networks Increases With Input Dimension (2018, methodology): “For all experiments, we only consider adversarial vulnerability to `∞-attacks, which we approximate using the FGSMimplementation of the python Foolbox-package (Rauber et al., 2017).”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “Cleverhans [139] and Foolbox [140] are open-source libraries to benchmark the vulnerability of DNNs against adversarial images.”
Towards the first adversarially robust neural network model on MNIST (2018, methodology): “To make this bound as tight as possible we apply many different attacks and choose the best one for each sample and model combination (using the implementations in Foolbox v1.”
On Evaluating Adversarial Robustness (2019, methodology): “There are several frameworks such as CleverHans (Papernot et al., 2018) or Foolbox (Rauber et al., 2017) as well as websites9,10,11 which have been developed to assist in this process.”
Robustness properties of Facebook's ResNeXt WSL models (2019, methodology): “We are very grateful to the authors of Geirhos et al. (2019), Hendrycks and Dietterich (2019), Hendrycks et al. (2019), Mahajan et al. (2018), Rauber et al. (2017) for making their pre-trained models, code, and stimuli publicly available, without which this work would not have been possible.”
Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defenses (2018, methodology): “r on the validation set. Inception V3 takes images of size 299 299 as input, which are cropped from images of size 342 342. For experiments with DeepFool [14], we used the implementation from Foolbox [16], with a budget of 100 iterations. For the experiments with C&W, we ported the attack (origi”
On Adaptive Attacks to Adversarial Example Defenses (2020, methodology): “All our experiments rely on implementations in Foolbox (Rauber et al., 2017).”
secml: A Python Library for Secure and Explainable Machine Learning (2019, background): “With respect to other popular libraries that implement attacks almost solely against DNNs (Papernot et al., 2018; Rauber et al., 2017; Nicolae et al., 2018), secml also implements training-time poisoning attacks and computationally-eﬃcient test-time evasion attacks against many diﬀerent algorithms,…”
First-Order Adversarial Vulnerability of Neural Networks and Input Dimension (2018, methodology): “5% of the pixel-range (invisible to humans), with PGD-attacks from the Foolbox-package (Rauber et al., 2017).”