CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

By Hamel Husain, Hongqiu Wu, Tiferet Gazit, Miltiadis Allamanis, Marc Brockschmidt. 2019

1,434 citations recorded by the source corpus.

Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas. To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task. We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.

Subfields: Information Systems, Software, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

  1. Chun Xia
  2. Prem Devanbu
  3. Lingming Zhang
  4. Charles Sutton
  5. Graham Neubig
  6. Miltiadis Allamanis
  7. Daya Guo
  8. Nan Duan

In Software

  1. Chun Xia
  2. Lingming Zhang
  3. Prem Devanbu
  4. Vincent J. Hellendoorn
  5. Daya Guo
  6. Neel Sundaresan
  7. Nan Duan
  8. Alexey Svyatkovskiy

In Natural Language Processing

  1. Graham Neubig
  2. Luke S. Zettlemoyer
  3. Mohit Bansal
  4. Percy Liang
  5. Hannaneh Hajishirzi
  6. Caiming Xiong
  7. Jianfeng Gao
  8. Yejin Choi

In Evaluation & Benchmarks

  1. Hannaneh Hajishirzi
  2. Mohit Bansal
  3. Sebastian Riedel
  4. Luke S. Zettlemoyer
  5. Caiming Xiong
  6. Eric Wallace
  7. Pieter Abbeel
  8. Wen-tau Yih

Citations in the corpus