Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

By Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette-Choo, Nicholas Carlini. 2022

147 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2021); they have even leaked software code and other protected intellectual property (Ippolito et al., 2022).”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “While there are several reasonable definitions of memorization, we use the one from Carlini et al. (2021) as it has received considerable attention in the literature (Yoon & Lee, 2021; Huang et al., 2022; Ginart et al., 2022; Ippolito et al., 2022; Biderman et al., 2023).”
Foundation Models and Fair Use (2023, background): “For instance, Ippolito et al. (2022) showed that minimally modified style-transfer prompts can evade filters developed based on the verbatim match criterion.”
Data Portraits: Recording Foundation Model Training Data (2023, methodology): “More recent work analyzes the use of a Bloom filter to prevent verbatim copying of common n-grams from training sets [19].”
PaLM 2 Technical Report (2023, neutral): “, 2021) or output filtering (Ippolito et al., 2022) are employed.”
Emergent and Predictable Memorization in Large Language Models (2023, background): “[28] introduce an interference-time intervention that has a 100% success rate at preventing verbatim memorization, but note that their methodology is easily subverted, and does not fulfill the intention behind the term “memorization,” which is the tendency of models to learn entire samples during tr”
Extracting Training Data from Diffusion Models (2023, methodology): “, using CLIP scores) as opposed to focusing on exact verbatim repetition (although recent language modeling work has begun to explore approximate memorization as well [35]).”
Large Language Models Struggle to Learn Long-Tail Knowledge (2022, background): “Doing so has various limitations, as memorization can be harmful or beneficial even in non-verbatim cases (Ippolito et al., 2022).”
Analyzing Leakage of Personally Identifiable Information in Language Models (2023, background): “In practice, learning basically means that only a fraction of the training dataset is memorized [29] and that some memorization is likely necessary to achieve high utility [20].”