Concealed Data Poisoning Attacks on NLP Models

By Eric Wallace, Tony Zhao, Shi Feng, Sameer Singh. 2021

234 citations recorded by the source corpus.

Adversarial attacks alter NLP model predictions by perturbing test-time inputs. However, it is much less understood whether, and how, predictions can be manipulated with small, concealed changes to the training data. In this work, we develop a new data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input. For instance, we insert 50 poison examples into a sentiment model’s training set that causes the model to frequently predict Positive whenever the input contains “James Bond”. Crucially, we craft these poison examples using a gradient-based procedure so that they do not mention the trigger phrase. We also apply our poison attack to language modeling (“Apple iPhone” triggers negative generations) and machine translation (“iced coffee” mistranslated as “hot coffee”). We conclude by proposing three defenses that can mitigate our attack at some cost in prediction accuracy or extra human annotation.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Poisoning Language Models During Instruction Tuning (2023, methodology): “Wallace et al. (2020) 2020; Schuster et al., 2021; Yang et al., 2021) or directly manipulating the model weights (Kurita et al., 2020).”
Execution-Based Evaluation for Open-Domain Code Generation (2022, background): “We should always keep alert to concealing malicious code (Wallace et al., 2021) or generating code with security vulnerabilities (Verdi et al., 2020; Pearce et al., 2021).”
Unsolved Problems in ML Safety (2021, background): “If an adversary uploads a few carefully crafted poisoned images [29], code snippets [156], or sentences [194] to platforms such as Flickr, GitHub or Twitter, they can inject a backdoor into future models trained on that data [10].”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “Relatedly, data poisoning research studies how to adversarially introduce artifacts into training data to produce unwanted model behaviors (Nelson et al., 2008; Chan et al., 2020; Wallace et al., 2021).”
MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages (2022, background): “Nevertheless, with the advances in models capable of generating code from natural language inputs, we should be aware of the potentially harmful usage such as concealing malicious code (Wallace et al., 2020), or generating code with security vulnerabilities (Verdi et al., 2020; Pearce et al., 2021).”
Poisoning Web-Scale Training Datasets is Practical (2023, background): “, a poisoning attack [6]) suffices to introduce targeted mistakes in model behavior [14, 15, 60, 76].”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
Tool Learning with Foundation Models (2023, unknown): “Moreover, due to training on massive web data, foundation models are faced with long-lasting training-time security issues in deep learning, such as backdoor attacks (Kurita et al., 2020; Cui et al., 2022) and data poisoning attacks (Wallace et al., 2021).”
A Categorical Archive of ChatGPT Failures (2023, background): “Moreover, due to the absence of supervised training, LLMs are vulnerable to data poisoning, which can result in the injection of hateful speech aimed at a specific entity, company, group, or individual [30].”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “Empirical assessments have demonstrated that limiting the number of training epochs is a straightforward method for LMs to reduce the impact of data poisoning such as RoBERTa [167].”
Customizing Triggers with Concealed Data Poisoning (2020, unknown): “Crafting Poison Second Order Gradients Inner Loop Update: θt+1 ← θt − η∇θLtrain ( Dcl ∪ D (t) adv; θ ) (3) η: Learning rate Outer Loop Update: D adv ← D (t) adv − ηadv∇DadvLadv ( D adv; θt+1 ) (4) θt+1: Used as a proxy for the inner minimizer Question: This learning paradigm works well for vision bu”
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2021, background): “For example, one can find local (word-840 level) perturbations that lead to a certain adversarial841 outcome, if the perturbations are restricted to well-842 defined semantic categories (e.g., “blue” can be843 perturbed to any other color name) (Guo et al.,844 2021; Yuan et al., 2021).845”