Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models

By Kushal Tirumala, Aram H. Markosyan, Luke S. Zettlemoyer, Armen Aghajanyan. 2022

288 citations recorded by the source corpus.

Despite their wide adoption, the underlying training and memorization dynamics of very large language models is not well understood. We empirically study exact memorization in causal and masked language modeling, across model sizes and throughout the training process. We measure the effects of dataset size, learning rate, and model size on memorization, finding that larger language models memorize training data faster across all settings. Surprisingly, we show that larger models can memorize a larger portion of the data before over-fitting and tend to forget less throughout the training process. We also analyze the memorization dynamics of different parts of speech and find that models memorize nouns and numbers first; we hypothesize and provide empirical evidence that nouns and numbers act as a unique identifier for memorizing individual training examples. Together, these findings present another piece of the broader puzzle of trying to understand what actually improves as models get bigger.

Subfields: Artificial Intelligence, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Scaling & Learning Theory

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “[54] Kushal Tirumala, AramHMarkosyan, Luke Zettlemoyer, and Armen Aghajanyan.”
Augmented Language Models: a Survey (2023, background): “Memorization (Tirumala et al., 2022) combined with compositionality (Zhou et al., 2022) capabilities made LLMs able to execute various tasks such as language understanding or conditional and 1See e.g. https://blog.google/products/search/search-language-understanding-bert/…”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “, 2022) or papers which train (but do not release) custom models for their studies (Tirumala et al., 2022; Hernandez et al., 2022).”
Measuring Forgetting of Memorized Training Examples (2022, background): “[74] show that large language models forget examples seen early in training.”
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models (2022, background): “Interestingly, such replication behavior occurs even for models that are not overfitting to their training data [34,63].”
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2022, background): “Memorization Accuracy (MA) First proposed by Tirumala et al. (2022), Memorization Accuracy (MA) is defined as follows: MA(x) = ∑T−1 t=1 1{ argmax(pθ(·|x<t)) = xt} T − 1 (4) MA quantifies how much fθ has memorized the given token sequences and can be used to analyze the training dynamics of large…”
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy (2022, unknown): “For example, Tirumala et al. (2022) study a similar per-token definition called exact memorization and Kandpal et al. (2022) a document-level definition called perfect memorization .”
Baichuan 2: Open Large-scale Language Models (2023, background): “Understanding these dynamics is key to unraveling the inner working mechanism of large language models (Biderman et al., 2023a; Tirumala et al., 2022).”
Emergent and Predictable Memorization in Large Language Models (2023, background): “…(to the best of our knowledge), some forms of memorization are actually beneficial: we want large language models to memorize factual events and details to avoid “hallucinating” plausible-sounding but errant facts to unsuspecting users [Power et al., 2022, Cao et al., 2022, Tirumala et al., 2022b].”
On Provable Copyright Protection for Generative Models (2023, background): “” (Lee et al., 2022a; Tirumala et al., 2022; Carlini et al., 2022) show that memorization becomes worse with model size and data reuse.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, methodology): “Tools Augmented Language Models Beyond relying entirely on memorization (Tirumala et al., 2022), interacting with tools enhances the fidelity and potency of LLMs (Parisi et al.”
Analyzing Leakage of Personally Identifiable Information in Language Models (2023, background): “[56] study the effect of memorization across variables such as the dataset size, learning rate and model size.”