On Evaluating Adversarial Robustness

By Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian J. Goodfellow, Aleksander Madry, Alexey Kurakin. 2019

1,007 citations recorded by the source corpus.

Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect. We believe a large contributing factor is the difficulty of performing security evaluations. In this paper, we discuss the methodological foundations, review commonly accepted best practices, and suggest new methods for evaluating defenses to adversarial examples. We hope that both researchers developing defenses as well as readers and reviewers who wish to understand the completeness of an evaluation consider our advice in order to avoid common pitfalls.

Subfields: Artificial Intelligence, Molecular Biology, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Prior work often evaluates on synthetic distribution shifts to measure robustness [9, 32, 38].”
Natural Adversarial Examples (2019, background): “Aside from the known difficulties in evaluating `p robustness correctly [9, 8], Gilmer et al.”
Scaling Laws for Reward Model Overoptimization (2022, methodology): “…of adversarial examples in all kinds of neural networks [Szegedy et al., 2013, Lin et al., 2017, Ebrahimi et al., 2018, Dai et al., 2018], and proposed methods to measure and increase neural network robustness [Gu and Rigazio, 2014, Zheng et al., 2016, Carlini et al., 2019, Guo et al., 2021].”
Adversarial Example Games (2020, background): “The NoBox setting is a much more challenging setting than more traditional threat models, yet it is representative of many real-world attack scenarios, where the attacker cannot interact with the target model [13].”
SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems (2018, background): “Here, we focus on evaluating the more challenging and realistic threat model of a fully adaptive white-box adversary [4] that is aware of the presence of SentiNet, of its architecture and its inner workings.”
Theoretical evidence for adversarial robustness through randomization: the case of the Exponential family (2019, methodology): “In this paper, to ensure that our results are not biased by obfuscated gradients, we follow the principles of [1, 5] and evaluate our randomized networks with this technique.”
Fast is better than free: Revisiting adversarial training (2020, background): “Since then, several best practices have been proposed to mitigate this problem (Carlini et al., 2019).”
Transfer of Adversarial Robustness Between Perturbation Types (2019, methodology): “As suggested in (Carlini et al., 2019), we use more steps for evaluation than for adversarial training to ensure PGD converges.”
Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness (2019, background): “As discussed in [14, 32], approaches to detecting adversarial attacks need to be evaluated against the strongest possible attacks - adaptive whitebox attacks which have full knowledge of the detection scheme and actively seek to bypass it.”
Certified Adversarial Robustness with Additive Noise (2018, methodology): “To ensure the robustness of our approach is not due to gradient masking, we use the expectation of the gradient with respect to the randomization when estimating gradients, to ensemble over randomization and eliminate the effect of randomness, as recommended in [10, 38].”
Lucid: A Practical, Lightweight Deep Learning Solution for DDoS Attack Detection (2020, background): “The construction of defences robust to adversarial attacks is an open problem [68] and an aspect which we will further explore for LUCID.”
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, methodology): “Following [8], we test a stronger PGD attacker ( = 0.”
Smooth Adversarial Training (2020, methodology): “To avoid these evaluation pitfalls, we run a set of sanity tests for SAT, following the recommendations in [7].”
RobustBench: a standardized adversarial robustness benchmark (2020, background): “[17] suggest that one indicator of possible overestimation of robustness is when black-box attacks are more effective than white-box ones.”