Emergent and Predictable Memorization in Large Language Models

By Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, Edward Raf. 2023

192 citations recorded by the source corpus.

Memorization, or the tendency of large language models (LLMs) to output entire sequences from their training data verbatim, is a key concern for safely deploying language models. In particular, it is vital to minimize a model's memorization of sensitive datapoints such as those containing personal identifiable information (PII). The prevalence of such undesirable memorization can pose issues for model trainers, and may even require discarding an otherwise functional model. We therefore seek to predict which sequences will be memorized before a large model's full train-time by extrapolating the memorization behavior of lower-compute trial runs. We measure memorization of the Pythia model suite and plot scaling laws for forecasting memorization, allowing us to provide equi-compute recommendations to maximize the reliability (recall) of such predictions. We additionally provide further novel discoveries on the distribution of memorization scores across models and data. We release all code and data necessary to reproduce the results in this paper at https://github.com/EleutherAI/pythia

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “While there are several reasonable definitions of memorization, we use the one from Carlini et al. (2021) as it has received considerable attention in the literature (Yoon & Lee, 2021; Huang et al., 2022; Ginart et al., 2022; Ippolito et al., 2022; Biderman et al., 2023).”
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023, unknown): “Recent studies suggest that the remarkable performance of some LLMs might be rooted in their ability to memorize training data (Bender et al., 2021; Magar & Schwartz, 2022; Carlini et al., 2023; Biderman et al., 2023), rather than in generalization.”
Scaling Data-Constrained Language Models (2023, unknown): “Deduplication may have value not captured in our benchmark, such as reducing memorization [42, 37, 16, 10].”
Gemma 3 Technical Report (2025, unknown): “Large language models may produce near-copies of some text used in training (Biderman et al., 2023; Carlini et al., 2021, 2022; Ippolito et al., 2022; Nasr et al., 2023).”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, background): “Language models have the capability to memorize examples from their training datasets, potentially enabling malicious users to extract private information [8, 7, 9].”