Measuring Forgetting of Memorized Training Examples

By Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot and 1 more. 2022

150 citations recorded by the source corpus.

Machine learning models exhibit two seemingly contradictory phenomena: training data memorization, and various forms of forgetting. In memorization, models overfit specific training examples and become susceptible to privacy attacks. In forgetting, examples which appeared early in training are forgotten by the end. In this work, we connect these phenomena. We propose a technique to measure to what extent models"forget"the specifics of training examples, becoming less susceptible to privacy attacks on examples they have not seen recently. We show that, while non-convex models can memorize data forever in the worst-case, standard image, speech, and language models empirically do forget examples over time. We identify nondeterminism as a potential explanation, showing that deterministically trained models do not forget. Our results suggest that examples seen early when training with extremely large datasets - for instance those examples used to pre-train a model - may observe privacy benefits at the expense of examples seen later.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, methodology): “We plot the dataset distribution across groups in Fig.”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “Prior work on the dynamics of memorization is generally limited to a few models in isolation (Jagielski et al., 2022; Elazar et al., 2022) or papers which train (but do not release) custom models for their studies (Tirumala et al., 2022; Hernandez et al., 2022).”
Emergent and Predictable Memorization in Large Language Models (2023, result): “While this exact experimental set-up and context is novel, it is still in tension with existing literature [22, 29, 51].”
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models (2022, background): “Interestingly, such replication behavior occurs even for models that are not overfitting to their training data [34,63].”
Large Language Models Struggle to Learn Long-Tail Knowledge (2022, background): “, 2020) or designing a curriculum to minimize forgetting (Jagielski et al., 2023).”