Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models

By Martin Riddell, Ansong Ni, Arman Cohan. 2024

73 citations recorded by the source corpus.

While large language models have achieved remarkable performance on various code generation benchmarks, there have been growing concerns regarding potential contamination of these benchmarks as they may be leaked into pretraining and finetuning data. While recent work has investigated contamination in natural language generation and understanding tasks, there has been less extensive research into how data contamination impacts the evaluation of code generation, which is critical for understanding the robustness and reliability of LLMs in programming contexts. In this work, we perform a comprehensive study of data contamination of popular code generation benchmarks, and precisely quantify their overlap with pretraining corpus through both surface-level and semantic-level matching. In our experiments, we show that there are substantial overlap between popular code generation benchmarks and open training corpus, and models perform significantly better on the subset of the benchmarks where similar solutions are seen during training. We also conduct extensive analysis on the factors that affects model memorization and generalization, such as model size, problem difficulty, and question length. We release all resulting files from our matching pipeline for future research.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Armando Solar-Lezama
2. Koushik Sen
3. Naman Jain
4. Tianjun Zhang
5. Wen-Ding Li
6. Ion Stoica
7. Swarat Chaudhuri
8. Rishabh Singh

In Evaluation & Benchmarks

1. Armando Solar-Lezama
2. Naman Jain
3. Tianjun Zhang
4. Ion Stoica

In Code Generation & Program Synthesis

1. Armando Solar-Lezama
2. Koushik Sen
3. Rishabh Singh

Citations in the corpus

• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “For code, Riddell et al. (2024) use edit distance and AST-based semantic-similarity to detect contamination.”