Extracting Training Data from Large Language Models

By Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Ú. Erlingsson and 2 more. 2020

3,168 citations recorded by the source corpus.

It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. For example, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, background): “In some very large language models, which can verbatim memorize and recall large parts of their pre-training data (Carlini et al., 2021), eﬀorts have sometimes been made to ﬁlter occurrences of downstream instances from pre-training data, but this has not been performed for MMLU in the literature.”
High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “Generative models can also reveal their training data [5, 90], which is of great concern when the data contain sensitive or personal information and were collected without explicit consent.”
Training language models to follow instructions with human feedback (2022, background): “…produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously; for a thorough review we…”
LaMDA: Language Models for Dialog Applications (2022, background): “, [58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]), and inappropriately revealing personally identifying information (PII) from training data [73].”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “…et al., 2021), including bias in representations (Blodgett et al., 2020), proper handling (Rogers, 2021) and documentation (Bender & Friedman, 2018) of training data, privacy (Abadi et al., 2016b; Carlini et al., 2021), and environmental concerns (Strubell et al., 2019; Patterson et al., 2021).”
Improving language models by retrieving from trillions of tokens (2021, background): “At the same time, Carlini et al. (2021) demonstrate that large-scale language models can perfectly memorise parts of their training data, suggesting that enhancing models with retrieval may lead to further improvements.”
Challenges in Detoxifying Language Models (2021, background): “, 2021), privacy concerns (Carlini et al., 2020), and environmental impact (Strubell et al.”
Quantifying Memorization Across Neural Language Models (2022, methodology): “Finally, k -eidetic memorization (Carlini et al., 2020), is a useful deﬁnition for unprompted memorization, but less useful for tightly bounding memorization by prompting with training data (as we will do).”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, methodology): “We demonstrate that the membership inference methods from Carlini et al. (2021b) are correlated with the number of duplicates of a sequence in the training data.”
Competition-level code generation with AlphaCode (2022, background): “A common concern for large language models trained on large amounts of data is that they solve downstream problems by simply memorizing the training set (30, 31).”
Program Synthesis with Large Language Models (2021, background): “…it is now increasingly understood that large language models can learn undesirable (e.g. biased) behavior from unlabeled training data, e.g., Bender and Koller [2020] and Bender et al. [2021], or can reveal training data, as well as sensitive information in the training data [Carlini et al., 2020].”
Deduplicating Training Data Makes Language Models Better (2021, background): “The privacy risks of data memorization, for example the ability to extract sensitive data such as valid phone numbers and IRC usernames, are highlighted by Carlini et al. (2020).”
Evaluating Large Language Models Trained on Code (2021, background): “Similar to large language models, Codex models can learn patterns present in their training data (Carlini et al., 2021).”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “For example, privacy-related issues (Carlini et al., 2020), misin-formation (Levy et al., 2021), or radicalizing views in generated text could make a group more likely to be attributed to speciﬁc stereotypes (e.g., through action guided by misinformation) or end up with diminished opportunities…”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “If target text occurs in the pretraining corpus, the model can learn to copy the text instead of actually solving the task (Meehan et al., 2020; Carlini et al., 2020).”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Studies indicate that LMs might memorize or leak these details (Carlini et al., 2019, 2021), and under certain techniques, attackers can decode private data from model inferences (Li et al., 2022a; Pan et al., 2020; Song and Raghunathan, 2020).”