Towards Evaluating the Robustness of Neural Networks
By Nicholas Carlini, David A. Wagner. 2016
10,045 citations recorded by the source corpus.
Neural networks provide state-of-the-art results for most machine learning tasks. Unfortunately, neural networks are vulnerable to adversarial examples: given an input x and any target classification t, it is possible to find a new input x' that is similar to x but classified as t. This makes it difficult to apply neural networks in security-critical areas. Defensive distillation is a recently proposed approach that can take an arbitrary neural network, and increase its robustness, reducing the success rate of current attacks' ability to find adversarial examples from 95% to 0.5%.In this paper, we demonstrate that defensive distillation does not significantly increase the robustness of neural networks by introducing three new attack algorithms that are successful on both distilled and undistilled neural networks with 100% probability. Our attacks are tailored to three distance metrics used previously in the literature, and when compared to previous adversarial example generation algorithms, our attacks are often much more effective (and never worse). Furthermore, we propose using high-confidence adversarial examples in a simple transferability test we show can also be used to break defensive distillation. We hope our attacks will be used as a benchmark in future defense attempts to create neural networks that resist adversarial examples.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Sergey Levine
- Bernt Schiele
- Jianfeng Gao
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Bernt Schiele
- Xiaogang Wang
- Alan L. Yuille
- Jitendra Malik
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Matthias Hein
- Nicolas Papernot
- Cho-Jui Hsieh
- Huan Zhang
- Florian Tramèr
- Aleksander Madry
- Bo Li
Citations in the corpus
- Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects (2018, background): “, by synthesizing an entire image [29], a small patch [18, 13], a few pixels [8], or only a single pixel [40].”
- Adversarial Example Games (2020, methodology): “For constrained optimization problems the Carlini-Wagner (CW) attack [12] is a powerful iterative optimization scheme which introduces an attack objective designed to maximize the distance between the target class and the most likely adversarial class.”
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, methodology): “on: ReLU. Weight initialization distribution: N(0;0:1) for FC layers and 5) for convolutional layers. SGD with learning rate 0:01 trained for 20 epochs. This network architecture derives from [9] and [2], however we do not use any dropout (the reference network has a 0.5 dropout rate on the FC layer”
- Unsolved Problems in ML Safety (2021, background): “Current research largely focuses on the problem of “ (cid:96) p adversarial robustness,” [125, 30] where an adversary attempts to induce a misclassification but can only perturb inputs subject to a small p -norm constraint.”
- Advances and Open Problems in Federated Learning (2019, methodology): “s been a wide variety of work concerning attacks and defenses for various attack vectors, namely model update poisoning [69, 101, 97, 294, 21], data poisoning [63, 123, 368, 133], and evasion attacks [64, 377, 187, 90, 283]. As we will see, federated learning enhances the potency of many attacks, an”
- Certified Robustness to Adversarial Examples with Differential Privacy (2018, background): “Attacks have used various norms to quantify an attacker’s ability to perturb an image: the 0-norm [43], 1-norm [28], [7], 2-norm [53], [7], [2], and ∞-norm [19], [59], [7], [2], [35].”
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “This is witnessed by the work by Carlini and Wagner [15, 79], who exploited a similar idea to show that several recent defenses proposed against minimally-perturbed adversarial examples are vulnerable to high-confidence ones, using a stronger attack similar to those proposed in our earlier work, and…”
- Deep Variational Information Bottleneck (2017, methodology): “Most attacks fall into three broad categories: optimization-based attacks (Szegedy et al., 2014; Carlini & Wagner, 2016; Moosavi-Dezfooli et al., 2016; Papernot et al., 2015; Robinson & Graham, 2015; Sabour et al., 2016), which directly run an optimizer such as L-BFGS or ADAM (Kingma & Ba, 2015) on…”
- Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “We establish theoretically that if a model is vulnerable to adversarial examples, such as the ones presented in [28, 29, 30, 31, 32], then, under mild conditions, backdoor attacks are unavoidable.”
- Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “d into two categories: evasion attacks and poisoning attacks [22]. Evasion attacks, also known as adversarial examples, perturb inputs at the test time to induce wrong predictions by the target model [5, 8, 15, 38, 56]. In contrast, poisoning attacks target the training process by maliciously modify”
- Adversarial Training and Robustness for Multiple Perturbations (2019, background): “, PGD [25] and our SLIDE attack with 100 steps and 40 restarts (20 restarts on CIFAR10), as well as Carlini and Wagner’s `2-attack [6] (C&W), and an `1-variant—EAD [8]; (1)Kang et al.”
- Adversarial Training for Free! (2019, methodology): “We attack all models using PGD attacks with K iterations on both the cross-entropy loss (PGD-K ) and the Carlini-Wagner loss (CW-K ) [Carlini and Wagner, 2017].”
- Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (2019, background): “These can be applied to DNNs during inference [38], [39], [40], [41], [42].”
- Wasserstein Adversarial Examples via Projected Sinkhorn Iterations (2019, background): “To name a few, distillation was proposed as a defense but was defeated (Papernot et al., 2016; Carlini & Wagner, 2017), realistic transformations seen by vehicles were thought to be safe until more robust adversarial examples were created (Lu et al., 2017; Athalye et al., 2018b), and many defenses…”
- Analyzing Federated Learning through an Adversarial Lens (2018, background): “Since the attack is targeted, the adversary also attempts to ensure that the global model converges to a point with good performance on the test or validation data We note that these inputs are not modified to induce misclassification as in the phenomenon of adversarial examples [5, 26].”
- Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, background): “Much work has been conducted on defending against adversarial examples or evasion attacks , in particular related to image data, in which an adversary would apply a small perturbation to an input of a classifier and achieve a wrong classification result (Carlini and Wagner, 2017).”
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples (2018, methodology): “Much of what we describe below has been discussed in prior work (Carlini & Wagner, 2017a; Madry et al., 2018); we repeat these points here and offer our own perspective for completeness.”