Universal Adversarial Triggers for Attacking and Analyzing NLP

By Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh. 2019

1,099 citations recorded by the source corpus.

Adversarial examples highlight model vulnerabilities and are useful for evaluation and interpretation. We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. We propose a gradient-guided search over tokens which finds short trigger sequences (e.g., one word for classification and four words for language modeling) that successfully trigger the target prediction. For example, triggers cause SNLI entailment accuracy to drop from 89.94% to 0.55%, 72% of “why” questions in SQuAD to be answered “to kill american people”, and the GPT-2 language model to spew racist output even when conditioned on non-racial contexts. Furthermore, although the triggers are optimized using white-box access to a specific model, they transfer to other models for all tasks we consider. Finally, since triggers are input-agnostic, they provide an analysis of global model behavior. For instance, they confirm that SNLI models exploit dataset biases and help to diagnose heuristics learned by reading comprehension models.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “Adversarial examples can be created for NLP models by inserting phrases (Jia and Liang, 2017; Wallace et al., 2019), paraphrasing questions (Ribeiro et al.”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “Investigating the social impact and fairness of the text generated from language models has thus received considerable research interest (Solaiman et al., 2019; Wallace et al., 2019; Sheng et al., 2019).”
Challenges in Detoxifying Language Models (2021, background): “Wallace et al. (2019) first demonstrated that synthetic text prompts can cause racist model continuations with GPT-2.”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, background): “In fact, other controllable generation techniques, including prompts (Wallace et al., 2019; Sheng et al., 2020; Shin et al., 2020) or conditional LMs (Keskar et al., 2019) will likely exhibit the same type of biases.”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “We additionally cross-reference publicly 14When releasing REALTOXICITYPROMPTS, we will include a flag for prompts belong to this challenging subset. available Reddit dumps15 to identify which subreddits the URLs were submitted to.”
Deduplicating Training Data Makes Language Models Better (2021, background): “This has implications far beyond metrics like per-plexity and validation loss, as learned models re-ﬂect the biases present in their training data (Bender et al., 2021; Wallace et al., 2019; Sheng et al., 2020).”
Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “Bias Mitigation For autocomplete and dialogue generation, Sheng et al. (2020) formulate bias triggers using gradient-based methods of Wallace et al. (2019).”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “In the absence of such methods, language models are known to demonstrate toxic/harmful behaviour (Sheng et al., 2019; Liang et al., 2021; Wallace et al., 2019), generate non-factual information (Maynez et al.”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, background): “[32] further showed that adversarial triggers [37] can be used to control biases in language generation.”
Recipes for Safety in Open-domain Chatbots (2020, background): “Wallace et al. (2019) show that certain "universal triggers" (provocative statements) can be used to prompt a language model to generate bad outputs.”
How Can We Know What Language Models Know? (2019, background): “…themselves can be used as a tool for text understanding by formulating queries in natural language and either generating textual answers directly (McCann et al., 2018; Radford et al., 2019), or assessing multiple choices and picking the most likely one (Zweig and Burges, 2011; Rajani et al., 2019).”
Mitigating harm in language models with conditional-likelihood filtration (2021, methodology): “We implemented universal adversarial triggers as per [Wallace et al., 2021] to generate potential filtration triggers from the learned distribution of the baseline model.”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, methodology): “Although the loss is a function of the entire sequence, here we adopt a greedy approach, similar to Ebrahimi et al. (2018); Wallace et al. (2019), in which we optimize for Table 5: Sentence samples in triplets, generated by {baseline GPT-2, PPLM-Discrim POSITIVE , PPLM-Discrim NEGATIVE },…”
Red Teaming Language Models with Language Models (2022, background): “, 2019, inter alia) and offensive text from LMs (Wallace et al., 2019; He and Glass, 2019; Liu et al., 2019; Song et al., 2020; Liu et al., 2020; Yu and Sagae, 2021).”
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021, background): “A range of recent works are based on the general idea of models revising their behavior according to changes in their input (Wallace et al., 2019; Gardner et al., 2020; Emelin et al., 2020; Ye and Ren, 2021; Schick and Schütze, 2020; Sheng et al., 2020).”
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, methodology): “Despite the impressive generation quality, these language models may produce toxic or biased content as found in many studies (Wallace et al., 2019; ∗Equal contribution.”
Poisoning Language Models During Instruction Tuning (2023, background): “Separately, another class of vulnerabilities is test-time adversarial examples, e.g., Wallace et al. (2019) also craft trigger phrases that cause errors but do so by searching for naturally-occurring trigger phrases.”
Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021, methodology): “initialized as [MASK] tokens and then iteratively updated, at each step using a gradient-based searching algorithm (Wallace et al., 2019) to replace one of the trigger tokens with the token that is estimated to maximize the likelihood of the gold label on the”