Deduplicating Training Data Mitigates Privacy Risks in Language Models

By Nikhil Kandpal, Eric Wallace, Colin Raffel. 2022

419 citations recorded by the source corpus.

Past work has shown that large language models are susceptible to privacy attacks, where adversaries generate sequences from a trained model and detect which sequences are memorized from the training set. In this work, we show that the success of these attacks is largely due to duplication in commonly used web-scraped training sets. We first show that the rate at which language models regenerate training sequences is superlinearly related to a sequence's count in the training set. For instance, a sequence that is present 10 times in the training data is on average generated ~1000 times more often than a sequence that is present only once. We next show that existing methods for detecting memorized sequences have near-chance accuracy on non-duplicated training sequences. Finally, we find that after applying methods to deduplicate training data, language models are considerably more secure against these types of privacy attacks. Taken together, our results motivate an increased focus on deduplication in privacy-sensitive applications and a reevaluation of the practicality of existing privacy attacks.

Subfields: Artificial Intelligence, Management Science and Operations Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, background): “In concurrent work, Kandpal et al. (2022) study how often models emit memorized data as a function of data duplication.”
Emergent Abilities of Large Language Models (2022, positive): “, 2021; 2022), though deduplication methods have been proposed and can simultaneously reduce memorization while improving performance (Kandpal et al., 2022; Lee et al., 2022a).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “Recent research has claimed to see significant benefits from going even further and deduplicating training data (Lee et al., 2021; Kandpal et al., 2022; Roberts et al., 2022).”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, neutral): “, 2021) Privacy Filtering private/duplicate data (Henderson et al., 2022; Kandpal et al., 2022; Lee et al., 2022b)”
Poisoning Language Models During Instruction Tuning (2023, background): “Other LM Vulnerabilites Similar to our work, past attacks also consider dangers of user-contributed data but focus on separate concerns such as privacy (Carlini et al., 2021; Kandpal et al., 2022).”
Foundation Models and Fair Use (2023, background): “In addition, exact or fuzzy data de-duplication based on the target semantic level is likely useful to attain the ideal benefit of the DP guarantee (Lee et al., 2021; Kandpal et al., 2022).”
The ROOTS Search Tool: Data Transparency for LLMs (2023, background): “…the ground truth to verify that model did have a chance to memorize some-thing that it should have memorized; • By providing match counts to identify which data was more likely to be memorized (since the number of copies in the training data inﬂu-ences memorization (Kandpal et al., 2022)).”
The Stack: 3 TB of permissively licensed source code (2022, background): “One could also have ethical concerns around the inclusion of Personally Identifiable Information (PII) and malicious code in the training data, as code LLMs might output such sensitive data (Carlini et al., 2021; Kandpal et al., 2022) or unsafe programs (Khlaaf et al., 2022) during deployment.”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “, 2021) and decreases memorization of training data (Kandpal et al., 2022).”
Counterfactual Memorization in Neural Language Models (2021, background): “There, the number of occurrences strongly correlate with the measured memorization [Carlini et al., 2020, Lee et al., 2021, Kandpal et al., 2022].”
Measuring Forgetting of Memorized Training Examples (2022, background): “Large language models memorize examples which are duplicated many times in their training sets [49, 41, 9].”
Do Language Models Plagiarize? (2022, methodology): “Likewise, prior memorization works [12, 25] documented the lower-bound of the plagiarism susceptibility and showed a small number of memorized instances.”
Provably Confidential Language Modelling (2022, background): “Recent works (Lee et al., 2022; Kandpal et al., 2022) show that deduplication enables language models to emit memorized text less frequently with same or better accuracy.”
Analyzing Leakage of Personally Identifiable Information in Language Models (2023, result): “[33] study models trained from scratch, whereas our study focuses on fine-tuned models and that further examination may be necessary to fully comprehend the relationship between our findings.”
Data Contamination: From Memorization to Exploitation (2022, background): “These works study the impact of duplication of training sequence on regeneration of the sequence (Carlini et al., 2022; Kandpal et al., 2022), and the effect on few-shot numerical reasoning (Razeghi et al., 2022).”
SemDeDup: Data-efficient learning at web-scale through semantic deduplication (2023, background): “[22] found that deduplication prevents memorization in LLMs and thus mitigates privacy concerns.”
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024, result): “Removing these duplicates (deduplicating) has been correlated with improvements in model performance [5] and a reduction in memorization of pretraining data [53, 54].”