Counterfactual Memorization in Neural Language Models

By Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, Nicholas Carlini. 2021

203 citations recorded by the source corpus.

Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data. Understanding this memorization is important in real world applications and also from a learning-theoretical perspective. An open question in previous studies of language model memorization is how to filter out"common"memorization. In fact, most memorization criteria strongly correlate with the number of occurrences in the training set, capturing memorized familiar phrases, public knowledge, templated texts, or other repeated data. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training. We identify and study counterfactually-memorized training examples in standard text datasets. We estimate the influence of each memorized training example on the validation set and on generated texts, showing how this can provide direct evidence of the source of memorization at test time.

Subfields: Artificial Intelligence, Cultural Studies, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, methodology): “Prior studies of memorization in language models either focus on models or datasets of a ﬁxed size (Carlini et al., 2019; Zhang et al., 2021; Thakkar et al., 2020) or identify a narrow memorization-versus-scale relationship (Carlini et al., 2020; Lee et al., 2021).”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Zhang et al. (2021) study this form of memorization in large LMs and find that training examples that are the most memorized are sufficiently different from other examples in the training set.”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “We also note that these benefits are not hypothetical, as a number of papers about the limits and ethics of LLMs has been explicitly enabled by the public release of previous models (Zhang et al., 2021; Kandpal et al., 2022; Carlini et al., 2022; Birhane et al., 2021; nostalgebraist, 2020; Meng et a”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Also, it is known that LLMs can suffer from problems in fairness, bias, privacy, and truthfulness Zhang et al. (2021); Lin et al. (2022); Liang et al. (2021); Bender et al. (2021).”
Foundation Models and Fair Use (2023, background): “…tend to suffer from difficulties in scaling due to high computational cost (e.g., leave-k-out retraining can be costly) (Feldman & Zhang, 2020; Zhang et al., 2021) or being inaccurate or erroneous when applied to complex but realistic model classes (Basu et al., 2020; Ghorbani et al., 2019;…”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “Data deduplication has become a key tool for language model projects following research showing that it both improves performance on downstream tasks (Lee et al., 2022; Zhang et al., 2021) and decreases memorization of training data (Kandpal et al., 2022).”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, methodology): “This framework is applied to study memorization in language models by Zhang et al. (2021), who ﬁnd that training examples that are memorized by the LM tend to have high inﬂuence of LM predictions on similar validation instances.”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, result): “While the setup is not directly comparable to previous works, similar trends have been observed in the literature (Carlini et al., 2021, 2022; Zhang et al., 2021).”
Detecting Pretraining Data from Large Language Models (2023, result): “This aligns with findings that LMs better memorize tail outliers (Feldman, 2020; Zhang et al., 2021).”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, result): “This is in contrast to parametric models, where removing high-risk data is infeasible after training (Bourtoule et al., 2020; Carlini et al., 2021) and data attribution at scale is difficult (Zhang et al., 2021; Han et al., 2023).”
Differentially Private Decoding in Large Language Models (2022, background): “Despite the performance gain, recent studies have shown that LLMs tend to unintentionally memorize portions of their training data (Zhang et al., 2021; Carlini et al., 2019, 2022).”
Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023, background): “Specifically for language models, a large body of work also studies the related phenomenon of memorization (Kandpal et al., 2022; Carlini et al., 2022b,a; Zhang et al., 2021), which enables membership inference and data extraction attacks in the first place.”
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy (2022, background): “…regurgitate potentially private information, like phone numbers and addresses, as well as memorize long sequences from their training sets (Carlini et al., 2019, 2021; Lee et al., 2021; Carlini et al., 2022; Zhang et al., 2021; Thakkar et al., 2021; Ramaswamy et al., 2020; Kandpal et al., 2022).”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, unknown): “In a different line of work motivated by estimating the influence of individual training examples, [9] defines a training example x as memorized if the difference in expected model performance (where model performance is defined as M(f) above) over subsets of data including x and subsets of data not”