Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong

By Warren He, James Wei, Xinyun Chen, Nicholas Carlini, Dawn Song. 2017

246 citations recorded by the source corpus.

Ongoing research has proposed several methods to defend neural networks against adversarial examples, many of which researchers have shown to be ineffective. We ask whether a strong defense can be created by combining multiple (possibly weak) defenses. To answer this question, we study three defenses that follow this approach. Two of these are recently proposed defenses that intentionally combine components designed to work well together. A third defense combines three independent defenses. For all the components of these defenses and the combined defenses themselves, we show that an adaptive adversary can create adversarial examples successfully with low distortion. Thus, our work implies that ensemble of weak defenses is not sufficient to provide strong defense against adversarial examples.

Subfields: Artificial Intelligence, Hardware and Architecture, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “A number of defenses have been proposed [43], [44], [45], [46], [47], yet many have shown to be less effective against an adaptive adversary [48], [49], [50], [51].”
Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (2018, background): “It is possible to bypass each defense independently (and ensembles of defenses usually are not much stronger than the strongest sub-component (He et al., 2017)).”
A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations (2017, background): “subsequently shown to be vulnerable to more sophisticated attacks [4, 14, 6].”
AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation (2018, background): “We observe that GSS and Ensemble defend the network in a way that makes it only slightly more provably robust, consistent with observations that these styles of defense are insufficient [16], [27].”
Generating Adversarial Examples with Adversarial Networks (2018, background): “Threat Model As shown in the literature, most of the current defense strategies are not robust when attacking against them [Carlini and Wagner, 2017b; He et al., 2017].”
Spatially Transformed Adversarial Examples (2018, background): “However, these defenses can either be evaded by C&W attacks or only provide marginal improvements (Carlini & Wagner, 2017b; He et al., 2017).”
The Robust Manifold Defense: Adversarial Training using Generative Models (2017, background): “This phenomenon is receiving a tremendous amount of recent attention (e.g. [31, 25, 48, 19, 22, 46] and references therein) for two good reasons: First a classiﬁer that can be easily fooled by non-perceivable noise poses a security threat in any real deployment.”
Certifying Some Distributional Robustness with Principled Adversarial Training (2017, background): “Consequently, researchers have proposed adversarial attack and defense mechanisms (Papernot et al., 2016a;b;c; Rozsa et al., 2016; Carlini & Wagner, 2017; He et al., 2017; Madry et al., 2017; Tramèr et al., 2017).”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, neutral): “After investigating several defensive approaches, [128]”
Adversarially Robust Generalization Requires More Data (2018, background): “[22] Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song.”
Theoretically Principled Trade-off between Robustness and Accuracy (2019, background): “In machine learning, study of adversarial defenses has led to significant advances in understanding and defending against adversarial threat (He et al., 2017).”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, methodology): “[76] also combined feature squeezing with the ensemble method proposed in [173] to show that strength of defenses does not”
On Evaluating Adversarial Robustness (2019, background): “More concerning than the fact that progress is slow is the fact that most proposed defenses are quickly shown to have performed incorrect or incomplete evaluations (Carlini & Wagner, 2016; 2017c; Brendel & Bethge, 2017; Carlini & Wagner, 2017a; He et al., 2017; Carlini & Wagner, 2017b; Athalye et al”
Evaluating and Understanding the Robustness of Adversarial Logit Pairing (2018, background): “Unfortunately, many proposed defenses have had their claims invalidated by new attacks within their corresponding threat models [4, 8, 5, 6, 2, 12, 1].”
Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality (2018, background): “However, these defenses can generally be evaded by Opt attacks, either wholly or partially (Carlini & Wagner, 2017a; He et al., 2017; Li & Vorobeychik, 2014; 2015).”
Towards Fast Computation of Certified Robustness for ReLU Networks (2018, background): “Indeed, many of these defensive mechanism have been shown to be ineffective when more advanced attacks are used (Carlini & Wagner, 2017c;a;b; He et al., 2017).”
With Great Training Comes Great Vulnerability: Practical Attacks against Transfer Learning (2018, background): “, successfully crafted adversarial samples that can fool an ensemble of models, by jointly optimizing misclassification objectives over all members of the ensemble [29].”
Improving the Robustness of Deep Neural Networks via Adversarial Training with Triplet Loss (2019, unknown): “However, it was shown that the ﬁrst type of defense methods had been broken through by various targeted countermeasures [Carlini and Wagner, 2017a; He et al. , 2017; Athalye et al. , 2018].”
CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition (2018, background): “In [51], Dawn Song attempted to combine multiple defenses including feature squeezing and the specialist+1 to construct a larger strong defense.”