Quantifying Memorization Across Neural Language Models

By Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, Chiyuan Zhang. 2022

944 citations recorded by the source corpus.

Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others). We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes more complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “…al., 2021; Bommasani & et. al., 2021), for instance, perpetuating or exacerbating social stereotypes and disparities that are reflected in training data (Sheng et al., 2021), memorizing and revealing private information (Carlini et al., 2022), or causing downstream harms (Weidinger et al., 2021).”
Solving Quantitative Reasoning Problems with Language Models (2022, background): “This is especially relevant as there has been much prior work indicating that language models often memorize some fraction of their training data (Trinh and Le, 2018; Radford et al., 2019; Carlini et al., 2022).”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Past and concurrent work uses similar definitions and experimental setups (McCoy et al., 2021; Lee et al., 2021; Carlini et al., 2022).”
Emergent Abilities of Large Language Models (2022, positive): “For extracting training data from language models, larger models were found to be more likely to memorize training data (Carlini et al., 2021; 2022), though deduplication methods have been proposed and can simultaneously reduce memorization while improving performance (Kandpal et al., 2022; Lee et…”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “We also note that these benefits are not hypothetical, as a number of papers about the limits and ethics of LLMs has been explicitly enabled by the public release of previous models (Zhang et al., 2021; Kandpal et al., 2022; Carlini et al., 2022; Birhane et al., 2021; nostalgebraist, 2020; Meng et a”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “…Raffel et al., 2020); (3) ML-based quality filtering , using lightweight models trained on known gold data to identify similar high-quality web documents (Wenzek et al., 2020; Brown et al., 2020); (4) deduplication , removing either exact duplicate spans or similar documents (Lee et al., 2022).”
Pretraining Language Models with Human Preferences (2023, background): “, 2022), but this strategy is also limited by the fact that large LMs are quite resistant to forgetting their training data (an effect that increases with model size; Carlini et al., 2022; Vu et al., 2022; Ramasesh et al., 2022).”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, background): “Combined with recent evidence that the memorization capacity of training data grows with model size (Magar and Schwartz, 2022; Carlini et al., 2022), the risk of”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “While there are several reasonable definitions of memorization, we use the one from Carlini et al. (2021) as it has received considerable attention in the literature (Yoon & Lee, 2021; Huang et al., 2022; Ginart et al., 2022; Ippolito et al., 2022; Biderman et al., 2023).”
TALM: Tool Augmented Language Models (2022, background): “Third, large models can memorize parts of their training data with undesirable consequences [Carlini et al., 2022].”
Foundation Models and Fair Use (2023, background): “Others have shown that foundation models can regurgitate training data (Carlini et al., 2019; Lee et al., 2022; Carlini et al., 2022; Kandpal et al., 2022; Carlini et al., 2021).”
Data Portraits: Recording Foundation Model Training Data (2023, background): “Further work [10] examined memorization and showed that it increased with scale.”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “While this kind of documentation is far from standard practice, it is becoming increasingly common given recent calls for better documentation (Rogers, 2021; Bender et al., 2021) as well as empirical studies on data memorization in language models (Carlini et al., 2019, 2022).”
MusicLM: Generating Music From Text (2023, methodology): “We adapt the methodology used in Carlini et al. (2022) to study the extent to which MusicLM might memorize music segments.”
Measuring Forgetting of Memorized Training Examples (2022, background): “Prior work has found that larger models generally memorize more [9], making this effect natural, but it is interesting that there is little difference between most of these”