Scalable Extraction of Training Data from (Production) Language Models

By Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. F. Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, Katherine Lee. arXiv.org, 2023

592 citations recorded by the source corpus.

This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT. Existing techniques from the literature suffice to attack unaligned models; in order to attack the aligned ChatGPT, we develop a new divergence attack that causes the model to diverge from its chatbot-style generations and emit training data at a rate 150x higher than when behaving properly. Our methods show practical attacks can recover far more data than previously thought, and reveal that current alignment techniques do not eliminate memorization.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/fc7ee1828030a818f52518022a39f6a3ada60222

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

StruQ: Defending Against Prompt Injection with Structured Queries (2024, unknown): “Beyond prompt injection and jail-breaking, researchers have studied other attacks on LLMs, including data extraction [52, 53, 54, 55, 56] and task-specific attacks to decrease the LLM’s performance [13, 57, 14].”
Gemma: Open Models Based on Gemini Research and Technology (2024, methodology): “We focus on discoverable memorization, which serves as a reasonable upper-bound on the memorization of a model (Nasr et al., 2023) and has been the common definition used in several studies (Anil et al., 2023; Carlini et al., 2022; Kudugunta et al., 2023).”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, background): “Pioneering work [27, 9, 41] has demonstrated that it is possible to extract private pre-training data from BERT [15], GPT-2 [43], and ChatGPT [1], respectively.”
Gemma 3 Technical Report (2025, unknown): “Large language models may produce near-copies of some text used in training (Biderman et al., 2023; Carlini et al., 2021, 2022; Ippolito et al., 2022; Nasr et al., 2023).”
Promises and challenges of generative artificial intelligence for human learning (2024, unknown): “They can generate text by taking inputs and repeatedly predicting the next word—a form of GenAI.”