NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

By Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier López de Lacalle, Eneko Agirre. 2023

382 citations recorded by the source corpus.

In this position paper, we argue that the classical evaluation on Natural Language Processing (NLP) tasks using annotated benchmarks is in trouble. The worst kind of data contamination happens when a Large Language Model (LLM) is trained on the test split of a benchmark, and then evaluated in the same benchmark. The extent of the problem is unknown, as it is not straightforward to measure. Contamination causes an overestimation of the performance of a contaminated model in a target benchmark and associated task with respect to their non-contaminated counterparts. The consequences can be very harmful, with wrong scientific conclusions being published while other correct ones are discarded. This position paper defines different levels of data contamination and argues for a community effort, including the development of automatic and semi-automatic measures to detect when data from a benchmark was exposed to a model, and suggestions for flagging papers with conclusions that are compromised by data contamination.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fei Huang
2. Binyuan Hui
3. Jingren Zhou
4. Sinan Tan
5. Xiaohuan Zhou
6. Shuai Bai
7. Kai Dang
8. Yunfei Chu

In Large Language Models

1. Fei Huang
2. Jingren Zhou
3. Shuai Bai
4. Yichang Zhang
5. Junyang Lin
6. Rui Men
7. Jianxin Ma

In Evaluation & Benchmarks

1. Fei Huang
2. Jingren Zhou
3. Sinan Tan
4. Shuai Bai
5. Kai Dang
6. Junyang Lin
7. Jinze Bai

Citations in the corpus

• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, neutral): “Existing works [6, 36, 48, 39] have suggested a growing risk of contamination, which undermines the reliability of benchmarks over time, motivating the need for benchmarks that are more frequently updated.”
• Qwen2 Technical Report (2024, background): “For large language models, what counts as contamination and how to run contamination analysis remain an active area of research (Ravaut et al., 2024; Golchin & Surdeanu, 2024; Sainz et al., 2023).”
• Don't Make Your LLM an Evaluation Benchmark Cheater (2023, neutral): “For evaluation, each submission is suggested to be accompanied with a special contamination analysis report. to be unconsciously triggered (Oren et al., 2023; Sainz et al., 2023).”
• Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems (2026, unknown): “Public benchmarks often end up in training data, intentionally or unintentionally through derivative datasets, and models trained on them obtain inflated scores (Matton et al., 2024; Sainz et al., 2023).”