Generating Natural Adversarial Examples

By Zhengli Zhao, Dheeru Dua, Sameer Singh. 2017

659 citations recorded by the source corpus.

Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed. Recent work on adversarial examples, i.e. inputs with minor perturbations that result in substantially different model predictions, is helpful in evaluating the robustness of these models by exposing the adversarial scenarios where they fail. However, these malicious perturbations are often unnatural, not semantically meaningful, and not applicable to complicated domains such as language. In this paper, we propose a framework to generate natural and legible adversarial examples that lie on the data manifold, by searching in semantic space of dense and continuous data representation, utilizing the recent advances in generative adversarial networks. We present generated adversaries to demonstrate the potential of the proposed approach for black-box classifiers for a wide range of applications such as image classification, textual entailment, and machine translation. We include experiments to show that the generated adversaries are natural, legible to humans, and useful in evaluating and analyzing black-box classifiers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Red Teaming Language Models with Language Models (2022, background): “We encourage future work to develop white-box red teaming methods, especially for generating more realistic adversarial examples (in the spirit of Zhao et al., 2018); white-box methods are disproportionately useful to internal red teams.”
Adversarial Example Games (2020, background): “Similar in spirit to our approach many attacks strategies benefit from employing a latent space to craft attacks [76, 68, 9].”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “More informative human studies evaluate grammaticality or similarity of the adversarial examples to the original ones (Zhao et al., 2018c; Alzantot et al., 2018).”
Counterfactual Fairness in Text Classification through Robustness (2018, methodology): “These methods include translating sentences into another language and back to generate semantically equivalent adversarial examples [19], using attributions to identify tokens or phrases that receive an inappropriate amount of importance (either too much or too little) [17], generating more natural ”
Semantically Equivalent Adversarial Rules for Debugging NLP models (2018, background): “Other paraphrase limitations: Paraphrase models based on neural machine translation are biased towards maintaining the sentence structure, and thus do not produce certain adversaries (e.g. Table 5b), which recent work on paraphrasing (Iyyer et al., 2018) or generation using GANs (Zhao et al., 2018) ”
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets (2019, background): “These challenges have been used as evidence that current systems are brittle (Belinkov and Bisk, 2018; Mudrakarta et al., 2018; Zhao et al., 2018; Glockner et al., 2018; Ebrahimi et al., 2018; Ribeiro et al., 2018, 1 Often referred to as “adversarial datasets” or “attacks”.”
On the Art and Science of Machine Learning Explanations (2018, background): “While residual and sensitivity analysis typically play a role in model debugging exercises, readers are encouraged to investigate newer methods such as anchors and burgeoning work in adversarial examples [24], [32].”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, methodology): “[68] utilized generative adversarial networks (GANs) as part of their approach to generate adversarial examples of images and texts, which made adversarial examples more natural to human.”
HotFlip: White-Box Adversarial Examples for Text Classification (2017, background): “Interest in understanding vulnerabilities of NLP systems is growing (Jia and Liang, 2017; Zhao et al., 2018; Belinkov and Bisk, 2018; Iyyer et al., 2018).”
Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering (2018, neutral): “, 2018), or NLI premises (Zhao et al., 2018).”
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification (2019, background): “, 2018), textual entailment (Zhao et al., 2018; Glockner et al., 2018), and text classification (Alzantot et al.”
Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models (2020, background): “, 2014; Chen, 2018), here the input search space is discrete and thus the traditional gradient-based optimization methodologies cannot be harnessed effectively (Zhang et al., 2019; Zhao et al., 2017).”
FreeLB: Enhanced Adversarial Training for Natural Language Understanding (2019, background): “ristics for creatingadversarialexamplesin theblack-boxsetting, oronspeciﬁc tasks. Jia & Liang(2017)propose to add distracting sentences to the input document in order to induce mis-classiﬁcation. Zhao et al. (2018) generate text adversaries by projecting the input data to a latent space using GA”
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations (2019, background): “This is more challenging than the adversarial setting commonly addressed in sequence-to-sequence models, where the objective is generating sequences characterized by the presence or absence of certain given tokens [8, 33].”
Semantic Adversarial Examples (2018, background): “In [21], the authors considered the problem of generating natural adversarial examples using generative adversarial networks.”
Constructing Unrestricted Adversarial Examples with Generative Models (2018, background): “[45] proposes mapping the input image to a latent space using GANs, and search for adversarial examples in the vicinity of the latent code.”
Disentangling Adversarial Robustness and Generalization (2018, methodology): “also tried on real datasets [93, 11, 110], rendering the manifold assumption questionable.”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, background): “In addition, the stochastic gradient part can be replaced by other types of adversarial examples, such as natural [40] and spatially transformed adversary [35].”