Rethinking Benchmark and Contamination for Language Models with Rephrased Samples

By Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica. 2023

215 citations recorded by the source corpus.

Large language models are increasingly trained on all the data ever produced by humans. Many have raised concerns about the trustworthiness of public benchmarks due to potential contamination in pre-training or fine-tuning datasets. While most data decontamination efforts apply string matching (e.g., n-gram overlap) to remove benchmark data, we show that these methods are insufficient, and simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures. Furthermore, we demonstrate that if such variation of test data is not eliminated, a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4. We validate such observations in widely used benchmarks such as MMLU, GSK8k, and HumanEval. To address this growing risk, we propose a stronger LLM-based decontamination method and apply it to widely used pre-training and fine-tuning datasets, revealing significant previously unknown test overlap. For example, in pre-training sets such as RedPajama-Data-1T and StarCoder-Data, we identified that 8-18\% of the HumanEval benchmark overlaps. Interestingly, we also find such contamination in synthetic dataset generated by GPT-3.5/4, suggesting a potential risk of unintentional contamination. We urge the community to adopt stronger decontamination approaches when using public benchmarks. Moreover, we call for the community to actively develop fresh one-time exams to evaluate models accurately. Our decontamination tool is publicly available at https://github.com/lm-sys/llm-decontaminator.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, background): “Existing works [6, 36, 48, 39] have suggested a growing risk of contamination, which undermines the reliability of benchmarks over time, motivating the need for benchmarks that are more frequently updated.”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “While previous works have attempted decontamination using both exact and fuzzy matches (Li et al., 2023b,d), it can be a non-trivial task (Team, 2024) and can be evaded using simple strategies like rephrasing (Yang et al., 2023).”
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “To conclude the best practice, We will explore the evaluation results with multiple approaches like manual selection, hash-based deduplication [13] and LLM-based decontamination [65].”
LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models (2024, unknown): “Besides, those benchmarks are also susceptible to contamination during the training time[79].”
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools (2024, result): “Additionally, previous works have reported HumanEval-contaminated training data [28; 19; 50] in own or other LLMs, making the results on HumanEval relatively less trustful than before.”
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024, background): “Secondly, the test sets in these benchmarks are static, meaning they can become contaminated over time, which undermines the reliability of the evaluation results (Yang et al., 2023).”
MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (2024, unknown): “Static benchmarks risk contamination over time as models may overfit to the benchmark data [6, 32, 36], undermining evaluation reliability.”
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024, background): “Such semantic comparison would be much harder and more prone to false-negatives for natural language ( i.e., the input), as illustrated by recent work (Yang et al., 2023).”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, unknown): “…and found that n-gram matching yielded the most useful results — while embedding-based methods can in principle identify non-trivial contamination like that due to paraphrasing [Yang et al., 2023], we found it difficult to distinguish mere distributional similarity from actual paraphrasing.”
Competition-Level Problems are Effective LLM Evaluators (2023, background): “This observation aligns with recent debates on large models’ data memorization tendencies (Carlini et al., 2023; Yang et al., 2023).”
Deep Literature Survey Automation with an Iterative Workflow (2025, unknown): “While these results confirm the superiority of IterSurvey, we find that absolute scoring struggles to reliably quantify the performance gap against human-written surveys (Yang et al., 2023b; Oren et al., 2023; Ye et al., 2024).”