DéjàVu: a map of code duplicates on GitHub

By Cristina Videira Lopes, Petr Maj, Pedro Martins, Vaibhav Pratap Singh Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, Jan Vitek. 2017

210 citations recorded by the source corpus.

Previous studies have shown that there is a non-trivial amount of duplication in source code. This paper analyzes a corpus of 4.5 million non-fork projects hosted on GitHub representing over 428 million files written in Java, C++, Python, and JavaScript. We found that this corpus has a mere 85 million unique files. In other words, 70% of the code on GitHub consists of clones of previously created files. There is considerable variation between language ecosystems. JavaScript has the highest rate of file duplication, only 6% of the files are distinct. Java, on the other hand, has the least duplication, 60% of files are distinct. Lastly, a project-level analysis shows that between 9% and 31% of the projects contain at least 80% of files that can be found elsewhere. These rates of duplication have implications for systems built on open source software as well as for researchers interested in analyzing large code bases. As a concrete artifact of this study, we have created DéjàVu, a publicly available map of code duplicates in GitHub repositories.

Subfields: Information Systems, Software, Program Analysis & Verification, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Program Analysis & Verification

In Signal Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Further, we remove duplicates based on Levenshtein distance between the files because duplicate files are known to be common in source code repositories (Lopes et al., 2017; Allamanis, 2019).”
The Stack: 3 TB of permissively licensed source code (2022, background): “Indeed, Lopes et al. (2017) observed that a large proportion of GitHub data consists of clones, resulting in a high ratio of exact and near-duplicates.”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, methodology): “on or toString in Java. •We remove duplicates from the dataset by identifying (near) duplicate functions and only keeping one copy of them (we use the methods described in Allamanis [1], Lopes et al. [18]). This removes multiple versions of auto-generated code and cases of copy & pasting. The fi”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “Indonesian News Corpus id Rahutomo and Miqdad Muadz Muzad (2018) IndoNLI id Mahendra et al. (2021) Indosum id Kurniawan and Louvan (2018) KALIMAT ar El-Haj and Koulali (2013) KSUCCA ar Alrabiah et al.”
code2vec: learning distributed representations of code (2018, methodology): “Reducing duplications Following recent work which found a large amount of code duplication in GitHub [Lopes et al. 2017], we took the top-ranked and most popular projects, in which we observed duplication to be less of a problem (Lopes et al.”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “[37] and Allamanis [3], we remove duplicate files.”
Learning to Generate Code Sketches (2021, methodology): “Then, we deduplicate the corpus using the method of Allamanis [1], Lopes et al. [20].”
Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python (2021, background): “On GitHub, Python projects often have file-level duplicates [28] and also code duplication has a negative effect on the performance of ML models when evaluating them on unseen code samples [1].”
Generative Code Modeling with Graphs (2018, methodology): “Dataset We have collected a dataset for our ExprGen task from 593 highly-starred open-source C# projects on GitHub, removing any near-duplicate files, following the work of Lopes et al. (2017). We parsed all C# files and identified all expressions of the fragment that we are considering (i.”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, methodology): “[42] and used in the construction of the CodeSearchNet database [25] to identify near-duplicates in our fine-tuning dataset.”
Multilingual training for Software Engineering (2021, background): “Sadly, there is a large amount of copied code on GitHUB [46]; inattentively combining different datasets harvested from GitHUB can lead to undesirable levels of duplication in the merged dataset.”
Typilus: neural type hints (2020, result): “Similar to the observations of Lopes et al. [44], we find a substantial number of (near) code duplicates in our corpus — more than 133k files.”
Neural Edit Completion (2020, methodology): “Following recent work on the adverse effects of code duplication [Allamanis 2019; Lopes et al. 2017], we split the dataset into training-validation-test by project .”
Structured Neural Summarization (2018, background): “This eludes to a potential issue, described by Lopes et al. (2017).”
Learning-Based Recursive Aggregation of Abstract Syntax Trees for Code Clone Detection (2019, background): “This problem is likely to become even more relevant in the future, due to today’s ease of online collaboration and code distribution via open-source code repositories [3], question-answering forums [4], and app stores [5].”