CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
By Hamel Husain, Hongqiu Wu, Tiferet Gazit, Miltiadis Allamanis, Marc Brockschmidt. 2019
1,434 citations recorded by the source corpus.
Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.
Subfields: Information Systems, Software, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Chun Xia
- Prem Devanbu
- Lingming Zhang
- Charles Sutton
- Graham Neubig
- Miltiadis Allamanis
- Daya Guo
- Nan Duan
In Software
- Chun Xia
- Lingming Zhang
- Prem Devanbu
- Vincent J. Hellendoorn
- Daya Guo
- Neel Sundaresan
- Nan Duan
- Alexey Svyatkovskiy
In Natural Language Processing
- Graham Neubig
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Hannaneh Hajishirzi
- Caiming Xiong
- Jianfeng Gao
- Yejin Choi
In Evaluation & Benchmarks
- Hannaneh Hajishirzi
- Mohit Bansal
- Sebastian Riedel
- Luke S. Zettlemoyer
- Caiming Xiong
- Eric Wallace
- Pieter Abbeel
- Wen-tau Yih
Citations in the corpus
- Program Synthesis with Large Language Models (2021, background): “…tasks including code completion [Raychev et al., 2016, Allamanis and Sutton, 2013b], clone detection [Svajlenko et al., 2014], code search [Husain et al., 2019], predicting readable names to describe functions [Allamanis et al., 2016], and generating function text from docstrings [Iyer et…”
- Evaluating Large Language Models Trained on Code (2021, methodology): “…towards the longstanding challenge of program synthesis (Simon, 1963; Manna & Waldinger, 1971), spurred by the presence of code in large datasets (Husain et al., 2019; Gao et al., 2020) and the resulting programming capabilities of language models trained on these datasets (Wang & Komatsuzaki,…”
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “CodeXGLUE cloze is created from CodeSearchNet to evaluate CodeBERT and consists of a short natural language description followed by code in several programming languages.”
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “We train CodeT5 on existing code corpus including CodeSearchNet (Hu-sain et al., 2019) and a small fraction of Google BigQuery, both of which are originally collected from public Github repositories.”
- Unified Pre-training for Program Understanding and Generation (2021, methodology): “While GPT-2 (Radford et al., 2019) is pretrained on NL corpora, CodeGPT-2 and CodeGPT-adapted are pretrained on CodeSearchNet (Lu et al., 2021).”
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “CodeXGLUE includes eight previously proposed datasets — BigCloneBench [65], POJ-104 [46], Devign [91], PY150 [57], Github Java Corpus [1], Bugs2Fix [69], CONCODE [34], and CodeSearchNet [32]— but also newly introduced datasets that are marked in Table 1.”
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages (2020, methodology): “The first four rows are reported by Husain et al. (2019), which are joint embeddings of NL and PL (Gu et al., 2018; Mitra et al., 2018).”
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, methodology): “…change prediction (Karaivanov, Raychev, and Vechev 2014; Oda et al. 2015; Barone and Sennrich 2017; Chen, Liu, and Song 2018; Kanade et al. 2019; Husain et al. 2019; Feng et al. 2020; Dinella et al. 2020; Lachaux et al. 2020), different automatic evaluation methods for code synthesis are…”
- PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers (2020, methodology): “CSN contains 500k P YTHON methods with docstrings, among 6 other languages.”
- Unit Test Case Generation with Transformers (2020, methodology): “For pretraining validation, we use the 239 test Java repositories from the CodeSearchNet [28], which comprise 600MB.”
- To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, methodology): “[42] and used in the construction of the CodeSearchNet database [25] to identify near-duplicates in our fine-tuning dataset.”
- A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “CodeSearchNet corpus (Husain et al., 2019) is built from GitHub projects in multiple mainstream programming languages.”
- Training and Evaluating a Jupyter Notebook Data Science Assistant (2022, background): “2020, 2019; Raychev, Vechev, and Yahav 2014; Bruch, Monperrus, and Mezini 2009), code search (Husain et al. 2019; Feng et al. 2020), bug fixing (Drain et al.”