Ensemble Adversarial Training: Attacks and Defenses

By Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, Patrick Drew McDaniel. 2017

3,060 citations recorded by the source corpus.

Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss. We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss. The model thus learns to generate weak perturbations, rather than defend against strong ones. As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step. We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models. On ImageNet, Ensemble Adversarial Training yields models with strong robustness to black-box attacks. In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks. However, subsequent work found that more elaborate black-box attacks could significantly enhance transferability and reduce the accuracy of our models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Among existing defense approaches, adversarial training (AT) is a general strategy to empower models with state-of-the-art robustness in different settings [65, 42, 72, 83, 50].”
Online Adversarial Attacks (2021, background): “Previously studied threat models (e.g., whitebox and blackbox) implicitly assume a static setting with full access to a target dataset [Tramèr et al., 2018].”
Adversarial Example Games (2020, methodology): “3 in [67]).”
Towards Deep Learning Models Resistant to Adversarial Attacks (2017, methodology): “While training against FGSM adversaries has shown some successes, recent work also highlights important shortcomings of this one-step approach [24].”
Unsolved Problems in ML Safety (2021, background): “If a blackbox ML system is not publicly released and can only be queried, it may be possible to practically defend the system against zero-query attacks [189] or limited-query attacks [34].”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, background): “[57] have also explored ensemble-based solutions to tackle adversarial examples, a particularly hard case of out-of-distribution examples.”
Advances and Open Problems in Federated Learning (2019, methodology): “0 or ImageNet [90]. In the black-box setting, models have also been shown to be vulnerable to attacks based on query-access to the model [99, 82] or based on substitute models trained on similar data [377, 315, 386]. While black-box attacks may be more natural to consider in datacenter settings, the”
Certified Robustness to Adversarial Examples with Differential Privacy (2018, background): “dients phenomenon, also referred to as gradient masking [45], [55], makes the defense susceptible to new attacks crafted to circumvent that obfuscation [2].”
Adversarial Training and Robustness for Multiple Perturbations (2019, background): “Training on weak attacks gives no robustness [40], while strong optimization attacks (e.”
Adversarial Training for Free! (2019, background): “The initial random noise was ﬁrst studied by Tramèr et al. [2017] to enable FGSM to attack models that rely on “gradient masking.””
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, background): “As (Tramèr et al., 2018; Schott et al., 2019) demonstrate, this binarizing classifier is highly `∞-robust, because most perturbations in the pixel space do not actually change the (thresholded) feature representation.”
Stronger data poisoning attacks break data sanitization defenses (2018, background): “Designing models that are robust to such attacks, as well as coming up with more effective attacks, is an active area of research (Papernot et al., 2016c; Madry et al., 2017; Tramèr et al., 2017; Wong and Kolter, 2018; Raghunathan et al., 2018; Sinha et al., 2018; Athalye et al., 2018; Papernot and ”
Spectral Signatures in Backdoor Attacks (2018, methodology): “We also point out that backdoor poisoning is related to adversarial examples [13, 26, 21, 12, 29, 4, 24, 32].”
Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (2018, methodology): “…defenses which cause gradient masking by learning to break gradient descent (e.g., by learning to make the gradients point the wrong direction (Tramèr et al., 2018)), we refer to the case where defenses are designed in such a way that the constructed defense necessarily causes gradient…”
A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations (2017, background): “[26] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick D.”
Attacking the Madry Defense Model with L1-based Adversarial Examples (2017, methodology): “For PGD without random starts, or I-FGM, the step size was set to be /40, which has been shown to be an effective attack setting in (Tramèr et al., 2017).”
ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models (2017, background): “Interested readers can refer to the recent works in [19, 27, 41, 44, 45] and the references therein.”