SourcererCC: Scaling Code Clone Detection to Big-Code
By Hitesh Sajnani, Vaibhav Pratap Singh Saini, Jeffrey Svajlenko, Chanchal Kumar Roy, Cristina Videira Lopes. 2015
641 citations recorded by the source corpus.
Despite a decade of active research, there has been a marked lack in clone detection techniques that scale to large repositories for detecting near-miss clones. In this paper, we present a token-based clone detector, SourcererCC, that can detect both exact and near-miss clones from large inter-project repositories using a standard workstation. It exploits an optimized inverted-index to quickly query the potential clones of a given code block. Filtering heuristics based on token ordering are used to significantly reduce the size of the index, the number of code-block comparisons needed to detect the clones, as well as the number of required token-comparisons needed to judge a potential clone. We evaluate the scalability, execution time, recall and precision of SourcererCC, and compare it to four publicly available and state-of-the-art tools. To measure recall, we use two recent benchmarks: (1) a big benchmark of real clones, BigCloneBench, and (2) a Mutation/Injection-based framework of thousands of fine-grained artificial clones. We find SourcererCC has both high recall and precision, and is able to scale to a large inter-project repository (25K projects, 250MLOC) using a standard workstation.
Subfields: Information Systems, Software, Natural Language Processing, Code Generation & Program Synthesis, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Miltiadis Allamanis
- Hongyu Zhang
- Baishakhi Ray
- Ge Li
- Michele Tufano
- Artur Andrzejak
- Lutz Büch
- Gabriele Bavota
In Software
- Miltiadis Allamanis
- Artur Andrzejak
- Lutz Büch
- Ge Li
- Hongyu Zhang
- Michele Tufano
- Gabriele Bavota
- Baishakhi Ray
In Natural Language Processing
- Miltiadis Allamanis
- Baishakhi Ray
- Hongyu Zhang
- Bowen Zhou
- Bing Xiang
- Gabriele Bavota
- Michele Tufano
- Ge Li
In Code Generation & Program Synthesis
In Signal Processing
Citations in the corpus
- The adverse effects of code duplication in machine learning models of code (2018, methodology): “To achieve this, we follow the general principles in SourcererCC [26], we tokenize each file and extract all identifier and literal tokens.”
- An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation (2018, methodology): “The only comparable tool is SourcererCC [70], which is above all better in terms of scalability, something not really relevant for our application scenario.”
- Learning-Based Recursive Aggregation of Abstract Syntax Trees for Code Clone Detection (2019, background): “The training supervision signal may be provided by any state-of-art code clone detector; here SourcererCC [38].”
- HELoC: Hierarchical Contrastive Learning of Source Code Representation (2022, background): “Codeclone detection, i.e., detecting whether two code snippets are similar or not, is a widely studied topic in software engineering research [44– 46].”
- Semantic code search via equational reasoning (2020, methodology): “General Code Search Classic code search is based on information retrieval techniques originally designed for textual search, which focus on the set (boolean model) or multiset (vector space model / “bag of words”) of terms which appear in the document [41, 73, 89].”
- DeepSim: deep learning code functional similarity (2018, methodology): “Existing code clone detectors mainly target type I-III code clones through measuring code syntactical similarity based on representations such as text [6, 7, 43], tokens [28, 36, 47], abstract syntax tree [10] or parse-tree [26].”
- VUDDY: A Scalable Approach for Vulnerable Code Clone Discovery (2017, methodology): “3) Function-level granularity: SourcererCC [19] attempts to detect Type-3 clones by using the bag-of-tokens technique.”
- GaDei: On Scale-Up Training as a Service for Deep Learning (2016, background): “Big code has seen great success in different areas of programing language and programming analysis, such like code completion[26, 44], type inference[45], and code similarity inspection[47, 54].”
- Cross-project code clones in GitHub (2018, methodology): “Sajnani et al. (2016) have developed a token-based clone detector that uses indexing techniques to find clones with high performance at large scale.”
- DéjàVu: a map of code duplicates on GitHub (2017, methodology): “The pipeline we used includes SourcererCC, a token-based code clone detection tool that is freely available and has been compared to other similar tools using those benchmarks [Sajnani 2016; Sajnani et al. 2016].”
- Retrieval-based Neural Source Code Summarization (2020, methodology): “Traditional code clone detection techniques such as DECKARD [25] and SourcererCC [47] detect similar code with the tree-based or token-based code comparison.”
- An Empirical Study of C++ Vulnerabilities in Crowd-Sourced Code Examples (2019, methodology): “We use the state-of-the-art clone detection tool, SourcererCC [47], to identify similar code between the Stack Overflow C++ code snippets.”
- Detecting Similar Programs via The Weisfeiler-Leman Graph Kernel (2016, methodology): “compression [5], malware similarity analysis [3], software theft detection [14], software maintenance [7], and Internet-scale code clone search [25].”
- Neural Detection of Semantic Code Clones Via Tree-Based Convolution (2019, methodology): “[7] H. Sajnani, V. Saini, J. Svajlenko, C. K. Roy, and C. V. Lopes, “SourcererCC: Scaling code clone detection to big-code,” in ICSE, 2016, pp. 1157–1168.”
- Supervised Deep Features for Software Functional Clone Detection by Exploiting Lexical and Syntactical Information in Source Code (2017, background): “CCFinderX [Kamiya et al., 2002] and SourcererCC [Sajnani et al., 2016] treat source codes as bags of tokens and compare subsequences to detect clones.”
- <inline-formula><tex-math notation="LaTeX">$\mu$</tex-math><alternatives><mml:math><mml:mi>μ</mml:mi></mml:math><inline-graphic xlink:href="zou-ieq1-2942930.gif"/></alternatives></inline-formula>VulDeePecker: A Deep Learning-Based System for Multiclass Vulnerability Detection (2019, neutral): “A popular approach is to use manually-defined patterns to detect vulnerabilities [4], [5], [6], [7], [8], [9], [10], which attained a limited success.”
- A Novel Neural Source Code Representation Based on Abstract Syntax Tree (2019, methodology): “Many software engineering methods, such as source code classification [1], [2], code clone detection [3], [4], [5], [6], defect prediction [7], [8] and code summarization [9], [10] have been proposed to improve software development and maintenance.”