Unrestricted Adversarial Examples

By Tom B. Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano, Ian J. Goodfellow. 2018

108 citations recorded by the source corpus.

We introduce a two-player contest for evaluating the safety and robustness of machine learning systems, with a large prize pool. Unlike most prior work in ML robustness, which studies norm-constrained adversaries, we shift our focus to unconstrained adversaries. Defenders submit machine learning models, and try to achieve high accuracy and coverage on non-adversarial data while making no confident mistakes on adversarial inputs. Attackers try to subvert defenses by finding arbitrary unambiguous inputs where the model assigns an incorrect label with high confidence. We propose a simple unambiguous dataset ("bird-or- bicycle") to use as part of this contest. We hope this contest will help to more comprehensively evaluate the worst-case adversarial risk of machine learning models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Natural Adversarial Examples (2019, methodology): “Our examples are closer in spirit to the recent thought experiment of an adversarial photographer discussed in Brown et al. [6], and by deﬁnition occur in the real world.”
Scalable agent alignment via reward modeling: a research direction (2018, background): “However, it is unclear how to deﬁne adversarial perturbation rigorously in a general way (Brown et al., 2018; Gilmer et al., 2018).”
Adversarial Training for High-Stakes Reliability (2022, background): “In contrast, we allow adversarial examples in an unrestricted space [32], with the goal of avoiding any feasible failures.”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “…recent work argued that the perturbation-based approach is unrealistically restrictive and called for the need of generalizing the concept of adversarial examples to the unrestricted case, including any input crafted to be misinterpreted by the learned model (Song et al., 2018; Brown et al., 2018).”
Constructing Unrestricted Adversarial Examples with Generative Models (2018, background): “For example, a contest [24] has recently been launched on unrestricted adversarial examples.”
Disentangling Adversarial Robustness and Generalization (2018, background): “Recently, generative models have also been utilized to craft – or learn – more natural adversarial examples [88, 11, 107, 80].”
Objective Robustness in Deep Reinforcement Learning (2021, background): “For these reasons, Brown et al. [8] introduced a contest to evaluate ML systems against unconstrained (as opposed to norm-constrained) adversaries.”
Regulatory Markets for AI Safety (2019, background): “With multiple regulators operating in the market, regulators might be incentivized or required to engage in contests such as those proposed by Brown et al. (2018).”
VALUATION : A N ADVERSARIAL APPROACH TO UNCOVER CATASTROPHIC FAILURES (2018, unknown): “In traditional adversarial examples, moving beyond norm ball specification has been difficult, because human evaluation may be necessary to determine ground truth labels (see Brown et al. (2018); Song et al. (2018)).”
Certified Adversarial Robustness with Additive Noise (2018, background): “More general definitions of adversarial examples are considered in some works [24], but we only address norm-bounded adversarial examples in this paper.”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “Nevertheless, there is also evidence of on-manifold adversarial examples [43], [313], [314].”
RobustBench: a standardized adversarial robustness benchmark (2020, background): “Some of these restrictions were also discussed by [12] for the warm-up phase of their challenge.”