Foolbox: A Python toolbox to benchmark the robustness of machine learning models

By Jonas Rauber, Wieland Brendel, Matthias Bethge. 2017

Subfields: Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “We also run gradient-free black-box attacks; the boundary attack (Brendel et al., 2017) and the brute-force pointwise attack (Rauber et al., 2017).”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “Cleverhans [139] and Foolbox [140] are open-source libraries to benchmark the vulnerability of DNNs against adversarial images.”
Effects of Degradations on Deep Neural Network Architectures (2018, background): “This perturbed images, known as adversarial examples, can be generated in various ways [17].”
Towards the first adversarially robust neural network model on MNIST (2018, methodology): “To make this bound as tight as possible we apply many different attacks and choose the best one for each sample and model combination (using the implementations in Foolbox v1.”
Robustness properties of Facebook's ResNeXt WSL models (2019, methodology): “We are very grateful to the authors of Geirhos et al. (2019), Hendrycks and Dietterich (2019), Hendrycks et al. (2019), Mahajan et al. (2018), Rauber et al. (2017) for making their pre-trained models, code, and stimuli publicly available, without which this work would not have been possible.”
A randomized gradient-free attack on ReLU networks (2018, methodology): “We apply both DeepFool , in the implementation of [20], and CW l 2 -attack.”
RobustBench: a standardized adversarial robustness benchmark (2020, background): “There are many libraries that focus primarily on implementa105 tions of popular adversarial attacks such as FoolBox [99], Cleverhans [95], AdverTorch [31], AdvBox 106 [43], ART [89], SecML [83].”
advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch (2019, methodology): “Compared to existing adversarial robustness related toolboxes (Papernot et al., 2016a; Rauber et al., 2017), advertorch aims for 1. simple and consistent APIs for attacks and defenses; 2. concise reference implementations, utilizing the dynamic computational graphs in PyTorch; and 3. fast…”
A Simple Cache Model for Image Recognition (2018, methodology): “The attacks were all implemented with Foolbox [14].”
Adversarial Robustness Against the Union of Multiple Perturbation Models (2019, methodology): “For `1 attacks, we use both the `1 PGD adversary as well as additional Foolbox implementations of `0 attacks at the same radius, namely the salt & pepper attack [Rauber et al., 2017] and the pointwise attack [Schott et al.”
On Adaptive Attacks to Adversarial Example Defenses (2020, methodology): “All our experiments rely on implementations in Foolbox (Rauber et al., 2017).”
FMix: Enhancing Mixed Sample Data Augmentation (2020, methodology): “…fast gradient sign method, standard gradient descent, projected gradient descent, additive uniform noise, and DeepFool (Moosavi-Dezfooli et al., 2016) attacks over the whole CIFAR-10 test set on PreAct-ResNet18 models subject to ∞ constraints using the Foolbox library (Rauber et al., 2020; 2017).”
AOGNets: Compositional Grammatical Architectures for Deep Learning (2017, methodology): “Top-1 accuracy comparisons under white-box adversarial attack using 1-step FGSM [15] with the Foolbox toolkit [48].”
Boundary Attack++: Query-Efficient Decision-Based Adversarial Attack (2019, methodology): “In Foolbox, the initialization is named as “BlendedUniformNoiseAttack,” and is used as the default initialization of Boundary Attack [28].”
Stateful Detection of Black-Box Adversarial Attacks (2019, methodology): “To compare directly with the NES attack, we use l∞ distance with the boundary attack (instead of the usual l2 distance) as implemented in FoolBox [33].”
Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack (2019, methodology): “We use DF from (Rauber et al., 2017), CW and EAD as in (Papernot et al.”
Gradient Starvation: A Learning Proclivity in Neural Networks (2020, background): “The disturbance on the input is computed by projected gradient descent (PGD) [84], a well-known adversarial attack.”