Generating Natural Language Adversarial Examples

By Moustafa Farid Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivastava, Kai-Wei Chang. 2018

1,038 citations recorded by the source corpus.

Deep neural networks (DNNs) are vulnerable to adversarial examples, perturbations to correctly classified examples which can cause the model to misclassify. In the image domain, these perturbations can often be made virtually indistinguishable to human perception, causing humans and state-of-the-art models to disagree. However, in the natural language domain, small perturbations are clearly perceptible, and the replacement of a single word can drastically alter the semantics of the document. Given these challenges, we use a black-box population-based optimization algorithm to generate semantically and syntactically similar adversarial examples that fool well-trained sentiment analysis and textual entailment models with success rates of 97% and 70%, respectively. We additionally demonstrate that 92.3% of the successful sentiment analysis adversarial examples are classified to their original label by 20 human annotators, and that the examples are perceptibly quite similar. Finally, we discuss an attempt to use adversarial training as a defense, but fail to yield improvement, demonstrating the strength and diversity of our adversarial examples. We hope our findings encourage researchers to pursue improving the robustness of DNNs in the natural language domain.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, background): “Neural networks are susceptible to adversarial examples—changes to the input which do or do not change the network’s prediction in surprising ways (Szegedy et al., 2014; Jia and Liang, 2017; Ribeiro et al., 2018; Alzantot et al., 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “Finally, Alzantot et al. (2018) developed an interesting population-based genetic algorithm for crafting adversarial examples for text classiﬁcation, by maintaining a population of modiﬁcations of the original sentence and evaluating ﬁtness of modiﬁcations at each generation.”
PAWS: Paraphrase Adversaries from Word Scrambling (2019, background): “Example perturbation rules include adding noise to inputs (Jia and Liang, 2017; Chen et al., 2018), word replacements (Alzantot et al., 2018; Ribeiro et al., 2018), and syntactic transformation (Iyyer et al., 2018).”
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification (2019, background): “…of state-of-the-art models for many natural language processing tasks, e.g. question answering (Jia and Liang, 2017; Chen et al., 2018; Ribeiro et al., 2018), textual entailment (Zhao et al., 2018; Glockner et al., 2018), and text classiﬁcation (Alzantot et al., 2018; Iyyer et al., 2018).”
Disentangling Adversarial Robustness and Generalization (2018, background): “Finally, adversarial examples have been applied to a wide variety of tasks, also beyond computer vision, e.g., [26, 18, 93, 47, 35, 54, 3, 16].”
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (2018, methodology): “Alzantot et al.(2018) proposed a genetic algorithm to attack sentiment analysis.”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, methodology): “These approaches use generative adversarial networks (Zhao et al., 2018b) and population-based optimization algorithms (Alzantot et al., 2018).”
DynaSent: A Dynamic Benchmark for Sentiment Analysis (2020, background): “These efforts seek to determine whether models have met specific learning targets (Alzantot et al., 2018; Glockner et al., 2018; Naik et al., 2018; Nie et al., 2019), exploit relatively superficial properties of their training data, (Jia and Liang, 2017; Kaushik and Lipton, 2018; Zhang et al.”
Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder (2020, background): “…are carried out by adding distracting phrases (Jia and Liang, 2017; Chan et al., 2018), editing the words and characters directly (Papernot et al., 2016; Alzantot et al., 2018; Ebrahimi et al., 2017) or paraphrasing sentences (Iyyer et al., 2018; Ribeiro et al., 2018; Mudrakarta et al., 2018).”
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP (2020, methodology): “…robustness of NLP models, including new methods for generating adversarial examples and better approaches to defending against these adversaries (Alzantot et al., 2018; Jin et al., 2019; Kuleshov et al., 2018; Li et al., 2019; Gao et al., 2018; Wang et al., 2019; Ebrahimi et al., 2017; Zang et…”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, neutral): “The threat models employed in this literature include synonym-based attacks [62–64], character-based substitutions [24], and spelling mistakes [65].”
Compositional Explanations of Neurons (2020, background): “Overall, these examples are reminiscent of the image-patch attacks of [9], adversarial NLI inputs [1, 37], and the data collection process for recent counterfactual NLI datasets [13, 22], but instead of searching among neuron visualizations or using black-box optimization to synthesize examples, we…”
RobustBench: a standardized adversarial robustness benchmark (2020, background): “, natural language processing [2], malware detection [51]) where adversarial robustness can be of interest.”
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency (2019, methodology): “In the area of Natural Language Processing (NLP), there is only a few lines of works done recently that address adversarial attacks for NLP tasks (Liang et al., 2018; Samanta and Mehta, 2017; Alzantot et al., 2018).”
Human-level play in the game of Diplomacy by combining language models with strategic reasoning (2022, background): “Much work has used adversarial or counterfactual examples to improve the robustness of natural language systems (30, 31).”
TextBugger: Generating Adversarial Text Against Real-world Applications (2018, methodology): “Meanwhile, replacing words with semantically and syntactically similar words can ensure that the examples are perceptibly similar [1].”
On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “Nevertheless, Alzantot et al. [4] first construct adversarial examples on sentiment analysis and textual entailment tasks, while Jin et al. [36] report that BERT can be evaded through natural language attacks.”