The adverse effects of code duplication in machine learning models of code

By Miltiadis Allamanis. 2018

352 citations recorded by the source corpus.

The field of big code relies on mining large corpora of code to perform some learning task towards creating better tools for software engineers. A significant threat to this approach was recently identified by Lopes et al. (2017) who found a large amount of near-duplicate code on GitHub. However, the impact of code duplication has not been noticed by researchers devising machine learning models for source code. In this work, we explore the effects of code duplication on machine learning models showing that reported performance metrics are sometimes inflated by up to 100% when testing on duplicated code corpora compared to the performance on de-duplicated corpora which more accurately represent how machine learning models of code are used by software engineers. We present a duplication index for widely used datasets, list best practices for collecting code corpora and evaluating machine learning models on them. Finally, we release tools to help the community avoid this problem in future research.

Subfields: Information Systems, Software, Natural Language Processing, Program Analysis & Verification, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

In Program Analysis & Verification

In Signal Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Further, we remove duplicates based on Levenshtein distance between the files because duplicate files are known to be common in source code repositories (Lopes et al., 2017; Allamanis, 2019).”
Competition-level code generation with AlphaCode (2022, background): “In code, data leakage and duplication across training and evaluation sets are particularly common (Allamanis, 2019), and many participants publish their solutions online after competitions.”
Deduplicating Training Data Makes Language Models Better (2021, background): “Allamanis (2019) shows that duplicate examples in code datasets cause worsened performance on code understanding tasks.”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “Recent work has shown that deduplicating training data can improve model performance and reduce the risk of memorizing training data [5, 43, 38].”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Deduplication has been identified as playing a significant role in improving language models (Allamanis, 2019; Lee et al., 2022).”
The Stack: 3 TB of permissively licensed source code (2022, methodology): “We largely follow the implementation10 of Allamanis (2019) and rely on a MinHash approach Broder 8https://blueoakcouncil.org/list 9See https://github.com/bigcode-project/bigcode-analysis/tree/main/data_analysis 10https://gist.github.com/mallamanis/ce1a3624b6d1a9ec9b6966e6b7181dcd (2000); Lee et al.…”
A systematic evaluation of large language models of code (2022, background): “10 Allamanis (2019) has shown that code duplication that commonly manifests in datasets of code adversely effects language modeling of code.”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, methodology): “• We remove duplicates from the dataset by identifying (near) duplicate functions and only keeping one copy of them (we use the methods described in Allamanis [1], Lopes et al.”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “Allamanis (2019) and Lopes et al. (2017) highlight the large fraction of near-duplicates present in code datasets and how they can inflate performance metrics.”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “It motivated other works studying deobfuscation of identifier names and identifier name proposal using n-grams (Allamanis et al., 2014; 2015), probabilistic models (Raychev et al., 2015; Bichsel et al., 2016; Vasilescu et al., 2017; Alon et al., 2018), and recurrent neural networks (Bavishi et al., ”
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation (2018, neutral): “ning and test set.Specifically, code duplication represents a significant threat to machine learning approaches on source code which could lead to inflated results, as recently described by Allamanis [2].In particular, there exists no pair < bfp i,bfp j > in our dataset (i.e., train, validatio”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “We also limited ourselves to three languages, and did not fully evaluate the impact of code duplication [1].”
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024, neutral): “This means that document-level deduplication techniques (e.g., Al-lamanis, 2019) cannot be used effectively, as other programs within the document may add too much noise.”
Learning type annotation: is big data enough? (2021, methodology): “We use de-duplication [2] to avoid risk of leakage from training to test.”
Deep Learning Based Vulnerability Detection: Are We There Yet? (2020, methodology): “Intra-set duplicates (sepcially duplicates inside test set) tend to artificially inflate the overall performance of a method [56], as evidenced by the discrepancy of the base-”
Contrastive Code Representation Learning (2020, background): “, 2019) and duplication (Allamanis, 2019) that degrade generalization.”
Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks (2021, methodology): “For effective machine learning, the data samples are expected to be independent and identically distributed (iid); otherwise, the resulting performance metric could be signiﬁcantly inﬂated [20].”
Structural Language Models for Any-Code Generation (2019, methodology): “This dataset was found to contain the least code duplication by Allamanis (2018b).”