Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
By Anish Athalye, Nicholas Carlini, David A. Wagner. 2018
3,560 citations recorded by the source corpus.
We identify obfuscated gradients, a kind of gradient masking, as a phenomenon that leads to a false sense of security in defenses against adversarial examples. While defenses that cause obfuscated gradients appear to defeat iterative optimization-based attacks, we find defenses relying on this effect can be circumvented. We describe characteristic behaviors of defenses exhibiting the effect, and for each of the three types of obfuscated gradients we discover, we develop attack techniques to overcome it. In a case study, examining non-certified white-box-secure defenses at ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 9 defenses relying on obfuscated gradients. Our new attacks successfully circumvent 6 completely, and 1 partially, in the original threat model each paper considers.
Subfields: Artificial Intelligence, Hardware and Architecture, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Sergey Levine
- Jianfeng Gao
- Ruslan Salakhutdinov
- Bernt Schiele
In Hardware and Architecture
- Matthias Hein
- Nicholas Carlini
- Francesco Croce
- J. Kolter
- Cho-Jui Hsieh
- Huan Zhang
- Tom Goldstein
- Florian Tramèr
In Safety, Robustness & Interpretability
- Matthias Hein
- Nicholas Carlini
- Aleksander Madry
- Florian Tramèr
- Nicolas Papernot
- Huan Zhang
- Cho-Jui Hsieh
- Francesco Croce
Citations in the corpus
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “Adversarial Training Adversarial machine learning is an active research area [63, 16, 5].”
- Adversarial Examples Are Not Bugs, They Are Features (2019, background): “In particular, there exist adversarial examples that violate the local linearity of the classifier [Mad+18], while classifiers that are less linear do not exhibit greater robustness [ACW18].”
- Scalable agent alignment via reward modeling: a research direction (2018, background): “…the effect of adversarially crafted inputs to neural networks (Szegedy et al., 2013), so far the empirically most effective strategy has been adversarial training : training the model explicitly on adversarially perturbed inputs (Madry et al., 2017; Uesato et al., 2018; Athalye et al., 2018).”
- Unsolved Problems in ML Safety (2021, background): “While defending against adversaries might seem to be a straightfoward problem, defenses are currently struggling to keep pace with attacks [8, 188], and much research is needed to discover how to fix these longstanding weaknesses.”
- Advances and Open Problems in Federated Learning (2019, background): “ks. Here, robustness is often measured by the model performance on white-box adversarial examples. Unfortunately, many proposed defenses have been shown to only provide a superficial sense of security [31]. On the other hand, adversarial training, in which a robust model is trained with adversarial e”
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “The fact that maximum- or high-confidence attacks are better suited to the task of security evaluation of learning algorithms (as well as to improve transferability across different models) is also witnessed by the work by Carlini and Wagner [16, 85] and follow-up work in [86, 87].”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “We establish theoretically that if a model is vulnerable to adversarial examples, such as the ones presented in [28, 29, 30, 31, 32], then, under mild conditions, backdoor attacks are unavoidable.”
- Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, methodology): “Following previous work [4, 33, 34, 61], the perturbation budget ε values are set to be 8/255, 0.”
- Adversarial Training and Robustness for Multiple Perturbations (2019, background): “This is an indication of gradient-masking [27, 40, 1].”
- Adversarial Training for Free! (2019, background): “While many defenses were broken by Athalye et al. [2018], PGD-based adversarial training was among the few that withstood strong attacks.”
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “A number of defenses have been proposed [43], [44], [45], [46], [47], yet many have shown to be less effective against an adaptive adversary [48], [49], [50], [51].”
- Wasserstein Adversarial Examples via Projected Sinkhorn Iterations (2019, background): “Some examples include wearing adversarial 3D printed glasses to fool facial recognition (Sharif et al., 2017), the use of adversarial graffiti to attack traffic sign classification (Eykholt et al., 2018), and printing adversarial textures on objects to attack image classifiers (Athalye et al., 2018b).”
- Stronger data poisoning attacks break data sanitization defenses (2018, background): “Designing models that are robust to such attacks, as well as coming up with more effective attacks, is an active area of research (Papernot et al., 2016c; Madry et al., 2017; Tramèr et al., 2017; Wong and Kolter, 2018; Raghunathan et al., 2018; Sinha et al., 2018; Athalye et al., 2018; Papernot and ”
- Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (2018, background): “A variety of defenses against adversarial inputs have been proposed [37,13] and broken [20,9,5]; research into defenses that provide strong guarantees of robustness is ongoing.”
- Ensemble Adversarial Training: Attacks and Defenses (2017, background): “…the accuracy of linear approximations of the model’s loss is as an instance of a gradient-masking phenomenon (Papernot et al., 2016b), which affects other defensive techniques (Papernot et al., 2016c; Carlini & Wagner, 2017a; Nayebi & Ganguli, 2017; Brendel & Bethge, 2017; Athalye et al., 2018).”