Poisoning Web-Scale Training Datasets is Practical

By Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, William W. Pearce, H. S. Anderson, Andreas F. Terzis, Kurt Thomas, Florian Tramèr. 2023

362 citations recorded by the source corpus.

Deep learning models are often trained on distributed, web-scale datasets crawled from the internet. In this paper, we introduce two new dataset poisoning attacks that intentionally introduce malicious examples to a model’s performance. Our attacks are immediately practical and could, today, poison 10 popular datasets. Our first attack, split-view poisoning, exploits the mutable nature of internet content to ensure a dataset annotator’s initial view of the dataset differs from the view downloaded by subsequent clients. By exploiting specific invalid trust assumptions, we show how we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD. Our second attack, frontrunning poisoning, targets web-scale datasets that periodically snapshot crowd-sourced content—such as Wikipedia—where an attacker only needs a time-limited window to inject malicious examples. In light of both attacks, we notify the maintainers of each affected dataset and recommended several low-overhead defenses.

Subfields: Artificial Intelligence, Toxicology, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, background): “Recent studies have demonstrated that web-scale datasets can be poisoned by introducing malicious data into a small percentage of samples [Carlini and Terzis, 2021, Carlini et al., 2023].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “However, a similar attack is successful for instruction tuning with very few examples [Wan et al., 2023, Xu et al., 2023a], and poisoning web-scale datasets is possible under realistic assumptions [Carlini et al., 2023a].”
Knowledge Conflicts for LLMs: A Survey (2024, neutral): “Recently, there has been growing appre-hension regarding the potential generation of mis-information by LLMs (Ayoobi et al., 2023; Kidd and Birhane, 2023; Carlini et al., 2023; Zhou et al., 2023c; Spitale et al., 2023; Chen and Shu, 2023b).”
DataComp: In search of the next generation of multimodal datasets (2023, methodology): “We also release a SHA256 hash of each image to guard against data poisoning in subsequent CommonPool downloads [18].”
Provable Robust Watermarking for AI-Generated Text (2023, neutral): “Additionally, the proliferation of synthetic data from LLMs poses challenges for training new models, as synthetic data needs to be detected and excluded before model training (Radford et al., 2022; Carlini et al., 2023).”
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (2024, neutral): “For example, an attacker can easily inject poisoned texts by maliciously editing Wikipedia pages [4]).”
Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation (2025, unknown): “Moreover, the robustness of multi-lingual RACG against adversarial threats such as data poisoning [34]–[36] presents a critical research frontier.”