Deep Text Classification Can be Fooled

By Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, Wenchang Shi. 2017

452 citations recorded by the source corpus.

In this paper, we present an effective method to craft text adversarial samples, revealing one important yet underestimated fact that DNN-based text classifiers are also prone to adversarial sample attack. Specifically, confronted with different adversarial scenarios, the text items that are important for classification are identified by computing the cost gradients of the input (white-box attack) or generating a series of occluded test samples (black-box attack). Based on these items, we design three perturbation strategies, namely insertion, modification, and removal, to generate adversarial samples. The experiment results show that the adversarial samples generated by our method can successfully fool both state-of-the-art character-level and word-level DNN-based text classifiers. The adversarial samples can be perturbed to any desirable classes without compromising their utilities. At the same time, the introduced perturbation is difficult to be perceived.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “Others computed gradients with respect to input word embeddings to identify and rank words to be modiﬁed (Samanta and Mehta, 2017; Liang et al., 2018).”
Synthetic and Natural Noise Both Break Neural Machine Translation (2017, background): “, 2015) or other techniques to find important text edit operations (Papernot et al., 2016b; Samanta & Mehta, 2017; Liang et al., 2017; Ebrahimi et al., 2017).”
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks (2018, neutral): “Liang et al., 2017).”
A Survey of Methods for Explaining Black Box Models (2018, methodology): “For example, Reference [60] shows that a nearly-invisible image perturbation can lead a NN to classify an object for another; Reference [81] constructs abstract synthetic images that are classified arbitrarily; Reference [72] finds deconstructed versions of an image that are indistinguishable from t”
Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models (2020, background): “Similarly, some work has focused on text classification (Kuleshov et al., 2018; Liang et al., 2018), but in the white-box setting as compared to our framework is proposed in the black-box setting.”
Negative Training for Neural Dialogue Response Generation (2019, background): “sentence classification, or seq2seq models) (Papernot et al., 2016; Samanta and Mehta, 2017; Liang et al., 2018; Ebrahimi et al., 2017; Belinkov and Bisk, 2017; Chen et al., 2017).”
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (2018, background): “METHODS GRADIENT BASED? WORD-LEVEL RNN? SEQUENTIAL OUTPUT? TARGETED ATTACK? EBRAHIMI ET AL [40] √ × √ CLASS JIA&LIANG [38] × √ × × LI ET AL [34] √ √ × CLASS PAPERNOT ET AL [33] √ × √ × GAO ET AL [37] × √ × BINARY SAMANTA&MEHTA [35] × × × BINARY ZHAO ET AL [39] / √ √ × LIANG ET AL [36] √ × × CLASS SE”
Detecting egregious responses in neural sequence-to-sequence models (2018, background): “no longer a one-hot vector), so usually gradient information is only utilized to help decide how to change the input for a better objective function value (Liang et al., 2018; Ebrahimi et al., 2017).”
Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-based Detection (2019, background): “In addition, adversarial text examples can also be used for attacking online review systems [19], [20].”
Is BERT Really Robust? Natural Language Attack on Text Classiﬁcation and Entailment (2019, background): “Especially, some works proposed to attack the targeted classifiers by replacing words in text with deliberately crafted misspelled words (Li et al., 2016; Liang et al., 2017; Gao et al., 2018; Li et al., 2018), which results in ungrammatical sentences.”
Say What I Want: Towards the Dark Side of Neural Dialogue Models (2019, background): “Text classification problems are studied in [30, 31]; sentiment analysis is involved in [32]; grammar error detection is investigated in [33].”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “TextFool [78] uses the concept of FGSM to approximate the contribution of text items that possess significant contribution to the text classification task.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “A few white-box attacks look for important text edit operations that will fool a classifier [99, 207, 268, 294].”
BERT: A Review of Applications in Natural Language Processing and Understanding (2021, methodology): “Existing methods of creating textual adversarial examples include misspelling words [39], removing individual words from text [40], deleting or inserting entire phrases [38], all of which generate artificial examples.”
Towards Crafting Text Adversarial Samples (2017, methodology): “We compare our method with the existing method, TextFool, as described in (9) using Twitter dataset.”
Review of Artificial Intelligence Adversarial Attack and Defense Technologies (2019, methodology): “In natural language processing fields, there are adversarial attacks in machine translation [29] and text generation [30].”
BAE: BERT-based Adversarial Examples for Text Classification (2020, background): “, 2018) or adding and deleting words (Li et al., 2016; Liang et al., 2017; Feng et al., 2018) for creating adversarial examples.”
Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data (2018, background): “Training Efficiency Success rate Black-box Saliency [23, 17] No High Medium No Projected FGSM [20] No High Low No Delete 1-score [16] No Low High Yes DeepWordBug No Low Medium Yes Greedy Attack No Low Highest Yes Gumbel Attack Yes High Medium Yes / No”
Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning (2020, background): “Dozens of attacks and defenses have been proposed [4, 10, 22, 29, 34, 46, 62, 63, 72, 73].”