Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples

By Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin-Yu Chen, Cho-Jui Hsieh. 2018

268 citations recorded by the source corpus.

Crafting adversarial examples has become an important technique to evaluate the robustness of deep neural networks (DNNs). However, most existing works focus on attacking the image classification problem since its input space is continuous and output space is finite. In this paper, we study the much more challenging problem of crafting adversarial examples for sequence-to-sequence (seq2seq) models, whose inputs are discrete text strings and outputs have an almost infinite number of possibilities. To address the challenges caused by the discrete input space, we propose a projected gradient method combined with group lasso and gradient regularization. To handle the almost infinite output space, we design some novel loss functions to conduct non-overlapping attack and targeted keyword attack. We apply our algorithm to machine translation and text summarization tasks, and verify the effectiveness of the proposed algorithm: by changing less than 3 words, we can make seq2seq model to produce desired outputs with high success rates. We also use an external sentiment classifier to verify the property of preserving semantic meanings for our generated adversarial examples. On the other hand, we recognize that, compared with the well-evaluated CNN-based classifiers, seq2seq models are intrinsically more robust to adversarial attacks.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Red Teaming Language Models with Language Models (2022, background): “In fact, prior work generally finds adversarial examples that appear arbitrary (e.g., changing a seemingly random character; Ebrahimi et al., 2018; Cheng et al., 2020) or unintelligible (“TH PEOPLEMan goddreams Blacks”; Wallace et al.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 2018a) White 3 Word MT (Cheng et al., 2018) White 3 Word MT, summarization (Mudrakarta et al.”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, background): “He and Glass (2019) and Cheng et al. (2018) do the same for text generation.”
Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness Against Adversarial Attack (2018, background): “In addition to image classification, attacks to other DNN-powered tasks have also been actively investigated, such as visual question answering [43, 1], image captioning [10], semantic segmentation [31, 1] and etc [12, 8, 39].”
Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models (2020, background): “Besides, unlike some related works (He and Glass, 2018; Cheng et al., 2018) where crafted text inputs are ungrammatical and meaningless, the inputs generated by our model are smooth and natural utterances, which is guaranteed by the language model.”
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations (2019, background): “The closest to this goal, Cheng et al. [8] requires the presence of certain tokens anywhere in the target sequence.”
Negative Training for Neural Dialogue Response Generation (2019, background): “, 2016), vulnerability to adversarial attacks (Cheng et al., 2018; Belinkov & Bisk, 2017), and the malicious (egregious) response problem (He & Glass, 2019).”
Detecting egregious responses in neural sequence-to-sequence models (2018, background): “machine translation or text summarization) (Cheng et al., 2018; Belinkov & Bisk, 2017) are vulnerable to adversarial attacks.”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, background): “applications [182] with limited research for MDS [28].”
TextBugger: Generating Adversarial Text Against Real-world Applications (2018, methodology): “…DNNs models have exhibited state-of-the-art performance in a lot of applications, recently they have been found to be vulnerable against adversarial examples which are carefully generated by adding small perturbations to the legitimate inputs to fool the targeted models [8, 13, 20, 25, 36, 37].”
Selective Question Answering under Domain Shift (2020, background): “Adversarially chosen ungrammatical text can also cause catastrophic errors (Wallace et al., 2019; Cheng et al., 2020).”
Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020, background): “These gradients are used to generate attacks that flip output words (Cheng et al., 2020), decode nonsense into arbitrary sentences (Chaturvedi et al.”
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation (2019, background): “Note that some existing adversarial perturbations such as syntactically controlled paraphrasing (Iyyer et al., 2018), exploiting backtranslation systems (Ribeiro et al., 2018), or using targeted keyword attack (Cheng et al., 2018) are beyond the speciﬁcation in this paper.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “…explored in computer vision (Zhang et al., 2018), and have more recently been generalized to the text domain (Miao et al., 2020; Chen et al., 2020c; Cheng et al., 2020b; Chen et al., 2020a) by performing interpolation between original data and token-level augmented data in the output space (Miao…”
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2019, background): “As shown by recent studies, there exists important problems other than exposure bias for the current NLG models, such as the likelihood trap (Holtz-man et al., 2020), factuality (Massarelli et al., 2019; He et al., 2021), or robustness (Cheng et al., 2018; Kassner and Schütze, 2019), etc.”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “Machine Translation [13, 24, 30, 156] TED Talks, WMT’16 Multimodal Translation Task”
On the Robustness of Self-Attentive Models (2019, methodology): “In the ﬁrst type of “targeted keyword at-tack” discussed in (Cheng et al., 2018), we attempt to generate an adversarial input sequence such that a speciﬁc keyword appears in the output sequence within the threshold ∆ of number of word changes we allowed.”
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP (2020, background): “(black-box) (Cheng et al., 2018) Nonoverlapping output Counter-fitted word embedding swap Greedy-WIR”
Is Robustness the Cost of Accuracy? - A Comprehensive Study on the Robustness of 18 Deep Image Classification Models (2018, background): “A common approach to evaluate the robustness of DNNs is via adversarial attacks [3,4,5,6,7,8,9,10,11], where imperceptible adversarial examples are crafted to mislead DNNs.”