Adversarial Risk and the Dangers of Evaluating Against Weak Attacks

By Jonathan Uesato, Brendan O'Donoghue, Aäron van den Oord, Pushmeet Kohli. 2018

663 citations recorded by the source corpus.

This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. The existence of adversarial examples in trained neural networks reflects the fact that expected risk alone does not capture the model's performance against worst-case inputs. We motivate the use of adversarial risk as an objective, although it cannot easily be computed exactly. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may be obscured to adversaries, by optimizing this surrogate rather than the true adversarial risk. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.

Subfields: Artificial Intelligence, Computer Networks and Communications, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…the effect of adversarially crafted inputs to neural networks (Szegedy et al., 2013), so far the empirically most effective strategy has been adversarial training : training the model explicitly on adversarially perturbed inputs (Madry et al., 2017; Uesato et al., 2018; Athalye et al., 2018).”
Adversarial Training for High-Stakes Reliability (2022, background): “Adversarial training for image classiﬁers Much recent work in adversarial training has been on preventing adversarial examples for image classiﬁers [22, 27].”
Adversarial Training for Free! (2019, background): “To put things further in perspective, Uesato et al. [2018] broke three defenses for (cid:15) = 2 non-targeted attacks on ImageNet [Guo et al., 2017, Liao et al., 2018, Xie et al., 2017], degrading their performance below 1%.”
Certified Adversarial Robustness via Randomized Smoothing (2019, background): “In fact, most heuristic defenses proposed in the literature were later “broken” by stronger adversaries (Carlini & Wagner, 2017; Athalye et al., 2018; Uesato et al., 2018; Athalye & Carlini, 2018).”
Training verified learners with learned verifiers (2018, background): “…has shown that many of the defense strategies proposed in the literature are really obfuscation strategies, that simply make it harder for gradient-based adversaries to attack the model by obfuscating gradients in the model and that these defenses are easily broken by stronger adversaries [2, 28].”
Evolving Robust Neural Architectures to Defend from Adversarial Attacks (2019, background): “are explained at [38]. still vulnerable to attacks [36]. Many recent variations of defenses were proposed [20], [11] [32], which are carefully analyzed and many of their shortcomings explained in [1],[37]. In this paper, different from previous approaches, we aim to tackle the robustness problems of”
Towards the first adversarially robust neural network model on MNIST (2018, background): “Reliably evaluating a model’s adversarial robustness is difficult, because robustness is easily overestimated when evaluating only against weak attacks [Uesato et al., 2018].”
Robustness May Be at Odds with Accuracy (2018, methodology): “In particular, many of the proposed robust training methods were subsequently shown to be ineﬀective (Carlini & Wagner, 2017; Athalye et al., 2018; Uesato et al., 2018).”
Theoretically Principled Trade-off between Robustness and Accuracy (2019, background): “Hence, it is natural to expect that some prior work on adversarial defense replaced the 0-1 loss 1(·) with a surrogate loss (Madry et al., 2018; Kurakin et al., 2017; Uesato et al., 2018).”
A Dual Approach to Scalable Verification of Deep Networks (2018, methodology): “While modiﬁcations to neural network training algorithms have been proposed to mitigate this phenomenonMadry et al. [2018], a comprehensive so-lution that is fully robust to adversarial attacks remains elusive [Carlini and Wagner, 2017b, Uesato et al., 2018].”
On Evaluating Adversarial Robustness (2019, background): “A key challenge of security evaluations is that while this adversarial risk (Madry et al., 2017; Uesato et al., 2018) is often computable in theory (e.g. with optimal attacks or brute force enumeration of the considered perturbations), it is usually intractable to compute exactly, and therefore in…”
Adversarial vulnerability for any classifier (2018, methodology): “Note that using the PGD adversarial training strategy of [10] (which constitutes one of the most robust models to date [15]), the robustness is significantly improved, despite still being ∼ 1 order of magnitude smaller than the baseline of 0.”
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty (2019, methodology): “We also observe that auxiliary self-supervision can provide gains when complemented with stronger defenses like TRADES [Zhang et al., 2019] and is not broken by gradient-free attacks such as SPSA [Uesato et al., 2018].”
Evaluating and Understanding the Robustness of Adversarial Logit Pairing (2018, background): “Evaluations done with weak attacks can be seen as giving loose bounds, while evaluations done with stronger attacks give tighter bounds of true adversarial risk [Uesato et al., 2018].”
Technical Report on the CleverHans v2.1.0 Adversarial Examples Library (2016, methodology): “Simultaneous perturbation stochastic approximation (SPSA) was introduced by Uesato et al. [24].”
On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses (2018, background): “This defense has also been evaluated by Uesato et al. (2018) and found to be ineffective.”
Are Labels Required for Improving Adversarial Robustness? (2019, methodology): “We run several additional checks against gradient masking [1, 45, 46], detailed in Appendix E.”
Unrestricted Adversarial Examples (2018, methodology): “We have selected three attacks for this warm-up, including two black-box pixel-wise optimization attacks [Uesato et al., 2018, Brendel et al., 2017] and a spatial attack [Engstrom et al., 2017].”
On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models (2018, background): “, obfuscating gradients against projected gradient attacks), and as such they are easily broken by stronger adversaries [11, 12].”