Proving Test Set Contamination in Black Box Language Models

By Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, Tatsunori B. Hashimoto. 2023

251 citations recorded by the source corpus.

Large language models are trained on vast amounts of internet data, prompting concerns and speculation that they have memorized public benchmarks. Going from speculation to proof of contamination is challenging, as the pretraining data used by proprietary models are often not publicly accessible. We show that it is possible to provide provable guarantees of test set contamination in language models without access to pretraining data or model weights. Our approach leverages the fact that when there is no data contamination, all orderings of an exchangeable benchmark should be equally likely. In contrast, the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of a canonically ordered benchmark dataset is significantly higher than the likelihood after shuffling the examples. We demonstrate that our procedure is sensitive enough to reliably prove test set contamination in challenging situations, including models as small as 1.4 billion parameters, on small test sets of only 1000 examples, and datasets that appear only a few times in the pretraining corpus. Using our test, we audit five popular publicly accessible language models for test set contamination and find little evidence for pervasive contamination.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Rethinking Benchmark and Contamination for Language Models with Rephrased Samples (2023, background): “Exchange detection (Oren et al., 2023) considers the order of test cases within a benchmark, suggesting that if a model remembers the sequence of test cases, it may be contaminated.”
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024, background): “Static benchmarks have certain issues, including contamination, saturation, overfitting, and a lack of human alignment (Yang et al., 2023; Oren et al., 2023).”
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024, unknown): “To search for contamination in these models, recent papers (Shi et al., 2023; Oren and Meister, 2023; Ranaldi et al., 2024; Deng et al., 2023) use the probabilities of model outputs to observe contamination.”
Don't Make Your LLM an Evaluation Benchmark Cheater (2023, background): “For evaluation, each submission is suggested to be accompanied with a special contamination analysis report. to be unconsciously triggered (Oren et al., 2023; Sainz et al., 2023).”
Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems (2026, unknown): “Because most training corpora are not public, such contamination is difficult to rule out after release (Oren et al., 2024).”
Deep Literature Survey Automation with an Iterative Workflow (2025, unknown): “While these results confirm the superiority of IterSurvey, we find that absolute scoring struggles to reliably quantify the performance gap against human-written surveys (Yang et al., 2023b; Oren et al., 2023; Ye et al., 2024).”
A Language Model from 1913: Pretraining on Historical Text (2026, unknown): “…has shown that modern LLMs can exhibit substantial performance degradation under relatively small perturbations or refor-mulations of popular reasoning benchmarks (Oren et al., 2024; Zhang et al., 2024, inter alia ), suggesting potential benchmark contamination or shortcut memorization effects.”