Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

By Ji Gao, Jack Lanchantin, Mary Lou Soffa, Yanjun Qi. 2018

869 citations recorded by the source corpus.

Although various techniques have been proposed to generate adversarial samples for white-box attacks on text, little attention has been paid to a black-box attack, which is a more realistic scenario. In this paper, we present a novel algorithm, DeepWordBug, to effectively generate small text perturbations in a black-box setting that forces a deep-learning classifier to misclassify a text input. We develop novel scoring strategies to find the most important words to modify such that the deep classifier makes a wrong prediction. Simple character-level transformations are applied to the highest-ranked words in order to minimize the edit distance of the perturbation. We evaluated DeepWordBug on two real-world text datasets: Enron spam emails and IMDB movie reviews. Our experimental results indicate that DeepWordBug can reduce the classification accuracy from 99% to 40% on Enron and from 87% to 26% on IMDB. Our results strongly demonstrate that the generated adversarial sequences from a deep-learning model can similarly evade other deep models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “Some studies imposed constraints on adversarial examples to have a small number of edit operations (Gao et al., 2018).”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, background): “Others have considered black-box adversarial examples for text classification (Gao et al., 2018) or NLP evaluation (Jia & Liang, 2017).”
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (2018, methodology): “This idea has been used in attacking sequence classiﬁcation models in Gong et al.(2018).”
L2-Nonexpansive Neural Networks (2018, background): “…of networks including image classiﬁcation (Carlini & Wagner, 2017a; Goodfellow et al., 2014), speech recognition (Kreuk et al., 2018; Alzantot et al., 2018; Carlini & Wagner, 2018), image captioning (Chen et al., 2017) and natural language processing (Gao et al., 2018; Ebrahimi et al., 2017).”
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023, positive): “We primarily adopt the settings from TextAttack for TextBugger and DeepWordBug, such as the repeat constraint which prohibits modifying words that have already been altered.”
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (2019, methodology): “…models (Szegedy et al., 2014; Goodfellow et al., 2015), including NLP in general (Papernot et al., 2016; Samanta and Mehta, 2017; Liang et al., 2018; Jia and Liang, 2017; Ebrahimi et al., 2018; Gao et al., 2018), and character-based NMT in particular (Heigold et al., 2018; Belinkov and Bisk, 2018).”
A Survey on Machine Learning Techniques for Cyber Security in the Last Decade (2020, background): “This modiﬁcation can change the whole sense of message and lead to a disaster [656].”
Generalizable Adversarial Attacks Using Generative Models (2019, background): “…while constrained optimization-based attack strategies are easily applicable to continuous input domains (i.e., images), adapting them to new modalities, such as discrete textual data [18, 7, 8] or graph-structured data that is non-i.i.d. [6, 38], represents a signiﬁcant challenge [35, 32].”
Towards Robust Toxic Content Classification (2019, background): “Previous work generating adversarial examples in text often assumes access to either the weights (Ebrahimi et al. 2018; Liang et al. 2017) or the raw prediction scores of the classifier (Li et al. 2018; Liang et al. 2018; Alzantot et al. 2018; Gao et al. 2018; Samanta and Mehta 2017).”
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP (2020, methodology): “(Gao et al., 2018) {Untargeted, Targeted} Classification Levenshtein edit distance {Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution}* Greedy-WIR”
Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Models (2018, background): “…2016; Narodytska and Kasiviswanathan, 2017; Liu et al., 2017; Carlini and Wagner, 2017; Papernot et al., 2017; Mironenco et al., 2017; Wong, 2017; Gao et al., 2018) and NLP ( Jia and Liang, 2017; Zhao et al., 2018; Belinkov and Bisk, 2018; Shalyminov et al., 2017; Mironenco et al., 2017; Iyyer…”
White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks (2019, background): “…for generating such examples include adding random or heuristically constructed noise (Belinkov and Bisk, 2017; Rodriguez and Rojas-Galeano, 2018; Gao et al., 2018), meaningpreserving modifications that change the surface form (Iyyer et al., 2018; Ribeiro et al., 2018), and human-in-the-loop…”
Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond (2020, methodology): “We build a three-layer MLP model with [64, 64, 10] neurons in each layer and conduct experiments using only 10% and 1% of the training data in MNIST and FashionMNIST, and we then test on the full test set to aggressively evaluate the generalization performance.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Others have considered black-box adversarial examples for text classification [118] or NLP evaluation [164].”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “These representative works received the interests of evaluation with adversarial examples recently [35, 54, 55, 92, 104, 112, 118, 130, 156].”
Contextualized Perturbation for Textual Adversarial Attack (2020, background): “Character-based models (Liang et al., 2019; Ebrahimi et al., 2018; Li et al., 2018; Gao et al., 2018, inter alia ) use misspellings to attack the victim systems; however, these attacks can often be defended by a spell checker (Pruthi et al., 2019; Vijayaraghavan and Roy, 2019; Zhou et al., 2019b;…”
Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data (2018, methodology): “[5] developed scoring functions applicable for sequence data, and proposed to modify characters of the features selected by the scoring functions.”