Poisoning Language Models During Instruction Tuning

By Alexander Wan, Eric Wallace, Sheng Shen, Daniel Klein. 2023

359 citations recorded by the source corpus.

Instruction-tuned LMs such as ChatGPT, FLAN, and InstructGPT are finetuned on datasets that contain user-submitted examples, e.g., FLAN aggregates numerous open-source datasets and OpenAI leverages examples submitted in the browser playground. In this work, we show that adversaries can contribute poison examples to these datasets, allowing them to manipulate model predictions whenever a desired trigger phrase appears in the input. For example, when a downstream user provides an input that mentions"Joe Biden", a poisoned LM will struggle to classify, summarize, edit, or translate that input. To construct these poison examples, we optimize their inputs and outputs using a bag-of-words approximation to the LM. We evaluate our method on open-source instruction-tuned LMs. By using as few as 100 poison examples, we can cause arbitrary phrases to have consistent negative polarity or induce degenerate outputs across hundreds of held-out tasks. Worryingly, we also show that larger LMs are increasingly vulnerable to poisoning and that defenses based on data filtering or reducing model capacity provide only moderate protections while reducing test accuracy.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

StarCoder: may the source be with you! (2023, background): “10 Social Impact and Limitations (Perez et al., 2022), adversarial testing (Wan et al., 2023), and/or by adding a robust safety layer (OpenAI, 2023b).”
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2023, background): “Several works (Mishra et al., 2022b; Chung et al., 2022; Wang et al., 2022b; Wan et al., 2023) have shown that larger instruction-tuned models are more easily steered than smaller and non-instruction-tuned models.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “However, a similar attack is successful for instruction tuning with very few examples [Wan et al., 2023, Xu et al., 2023a], and poisoning web-scale datasets is possible under realistic assumptions [Carlini et al., 2023a].”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “In this section, we examine the range of existing defense methods against various attacks and vulnerabilities associated with LLMs 1 .”
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, neutral): “Backdoor attacks embed triggers into the LLMs to generate noxious outputs (Cai et al., 2022; Wan et al., 2023; Li et al., 2024; Zhang et al., 2024a).”
Many-shot Jailbreaking (2024, background): “For example, analogous to data poisoning (Wan et al., 2023), in which the training data is covertly manipulated towards an end, an adversary could potentially inject poisonous content into the context window of an LLM (e.g. in a legitimate-looking textbook that’s blindly placed in an LLM’s…”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “Other attacks to LLM: Other attacks to LLMs (or LLM-Integrated Applications) include, but not limited to, privacy attacks [16, 30], poisoning attacks [13, 42, 47, 54], and adversarial prompts [56].”