When deep learning met code search

By José Pablo Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, Satish Chandra. 2019

239 citations recorded by the source corpus.

There have been multiple recent proposals on using deep neural networks for code search using natural language. Common across these proposals is the idea of embedding code and natural language queries into real vectors and then using vector distance to approximate semantic correlation between code and the query. Multiple approaches exist for learning these embeddings, including unsupervised techniques, which rely only on a corpus of code examples, and supervised techniques, which use an aligned corpus of paired code and natural language descriptions. The goal of this supervision is to produce embeddings that are more similar for a query and the corresponding desired code snippet. Clearly, there are choices in whether to use supervised techniques at all, and if one does, what sort of network and training to use for supervision. This paper is the first to evaluate these choices systematically. To this end, we assembled implementations of state-of-the-art techniques to run on a common platform, training and evaluation corpora. To explore the design space in network complexity, we also introduced a new design point that is a minimal supervision extension to an existing unsupervised technique. Our evaluation shows that: 1. adding supervision to an existing unsupervised technique can improve performance, though not necessarily by much; 2. simple networks for supervision can be more effective that more sophisticated sequence-based networks for code search; 3. while it is common to use docstrings to carry out supervision, there is a sizeable gap between the effectiveness of docstrings and a more query-appropriate supervision corpus.

Subfields: Artificial Intelligence, Information Systems, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Code Generation & Program Synthesis

Citations in the corpus

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, methodology): “For this, we follow other attempts in the literature [5, 6, 9, 11] and pair functions in open-source software with the natural language present in their respective documentation.”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “Machine learning is frequently used on tasks involving programming languages, including code completion [28, 33, 35, 42], bug detection and code repair [15, 38, 46], code summarization [8, 25], clone detection [2, 47, 48], code search [13, 23] and code translation [14, 41].”
An Empirical Evaluation of GitHub Copilot's Code Suggestions (2022, background): “While there is a lot of work on code synthesis and code search using deep learning (e.g., [3–5, 8, 12, 14, 19, 32]), we do not dive into the specifics here since we focus on evaluating Copilot’s capabilities while treating it as a black box.”
CoDesc: A Large Code–Description Parallel Dataset (2021, methodology): “We use Mean Reciprocal Rank (MRR) – the commonly used evaluation metric for code search (Husain et al., 2019; Sachdev et al., 2018; Cambronero et al., 2019) as the evaluation criteria for code search.”
Adversarial examples for models of code (2019, methodology): “2017], code search [Cambronero et al. 2019; Liu et al. 2019; Sachdev et al. 2018], and bug finding [Pradel and Sen 2018; Rice et al.”
Pre-trained Contextual Embedding of Source Code (2019, methodology): “, 2018) and to evaluate their utility to provide supervision in neural code search (Cambronero et al., 2019).”
Semantic code search via equational reasoning (2020, background): “Cambronero et al [6] survey many of these techniques.”