Detecting Pretraining Data from Large Language Models

By Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, Luke S. Zettlemoyer. International Conference on Learning Representations, 2023

424 citations recorded by the source corpus.

Although large language models (LLMs) are widely deployed, the data used to train them is rarely disclosed. Given the incredible scale of this data, up to trillions of tokens, it is all but certain that it includes potentially problematic text such as copyrighted materials, personally identifiable information, and test data for widely reported reference benchmarks. However, we currently have no way to know which data of these types is included or in what proportions. In this paper, we study the pretraining data detection problem: given a piece of text and black-box access to an LLM without knowing the pretraining data, can we determine if the model was trained on the provided text? To facilitate this study, we introduce a dynamic benchmark WIKIMIA that uses data created before and after model training to support gold truth detection. We also introduce a new detection method Min-K% Prob based on a simple hypothesis: an unseen example is likely to contain a few outlier words with low probabilities under the LLM, while a seen example is less likely to have words with such low probabilities. Min-K% Prob can be applied without any knowledge about the pretraining corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data. Moreover, our experiments demonstrate that Min-K% Prob achieves a 7.4% improvement on WIKIMIA over these previous methods. We apply Min-K% Prob to two real-world scenarios, copyrighted book detection, and contaminated downstream example detection, and find it a consistently effective solution.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/3422d5e0cdfdc935d6a84a1e3d3f96659265fe3a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ion Stoica
2. Joseph E. Gonzalez
3. Lianmin Zheng
4. Song Mei
5. Yu Bai
6. Licong Lin
7. Shuo Yang
8. Ruiqi Zhang

Citations in the corpus

• Rethinking Benchmark and Contamination for Language Models with Rephrased Samples (2023, methodology): “The min-k prob detection (Shi et al., 2023) uses outlier tokens to estimate LLM contamination.”
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning (2024, methodology): “Finally, we note the existence of attack methods for extracting data from unlearned models (Shi et al., 2023; Patil et al., 2023), and other unlearning methods including model editing (Mitchell et al., 2022; Meng et al., 2022) and in-context unlearning (Pawelczyk et al., 2023).”
• TOFU: A Task of Fictitious Unlearning for LLMs (2024, neutral): “For example, Eldan & Russinovich (2023) explore unlearning information about Harry Potter; while they show promising results Shi et al. (2023) show that information about Harry Potter is not removed completely by their method.”
• Black-box model classification under the discriminative factorization (2026, unknown): “Pretraining data detection [34, 22] requires either similar assumptions or explicit examples of data in the training set.”
• Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations (2025, unknown): “…accessibility assumptions – for example, it is possible to determine if a model was trained on a particular type of data when token-wise log-probabilities of a relevant set of tokens are available (Shi et al., 2023) as well as when users only have access to model responses (Helm et al., 2025).”