Neural Code Comprehension: A Learnable Representation of Code Semantics

By Tal Ben-Nun, Alice Shoshana Jakobovits, Torsten Hoefler. 2018

276 citations recorded by the source corpus.

With the recent success of embeddings in natural language processing, research has been conducted into applying similar methods to code analysis. Most works attempt to process the code directly or use a syntactic tree representation, treating it like sentences written in a natural language. However, none of the existing methods are sufficient to comprehend program semantics robustly, due to structural features such as function calls, branching, and interchangeable order of statements. In this paper, we propose a novel processing technique to learn code semantics, and apply it to a variety of program analysis tasks. In particular, we stipulate that a robust distributional hypothesis of code applies to both human- and machine-generated programs. Following this hypothesis, we define an embedding space, inst2vec, based on an Intermediate Representation (IR) of the code that is independent of the source programming language. We provide a novel definition of contextual flow for this IR, leveraging both the underlying data- and control-flow of the program. We then analyze the embeddings qualitatively using analogies and clustering, and evaluate the learned representation on three different high-level tasks. We show that even without fine-tuning, a single RNN architecture and fixed inst2vec embeddings outperform specialized approaches for performance prediction (compute device mapping, optimal thread coarsening); and algorithm classification from raw code (104 classes), where we set a new state-of-the-art.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Information Systems, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Information Systems

In Code Generation & Program Synthesis

Citations in the corpus

Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks (2021, methodology): “Two popular datasets of a similar kind are POJ-104 [16] and GCJ [17] (derived from Google Code Jam).”
Contrastive Code Representation Learning (2020, methodology): “Inst2vec (Ben-Nun et al., 2018) embeds statements in LLVM IR by processing a flow graph with a context prediction objective (Mikolov et al.”
MISIM: An End-to-End Neural Code Similarity System (2020, methodology): “Although research in the space of code similarity is still in the early stages, there is a growing body of exploratory work in this area [5,10,23,26,29].”
Modeling Vocabulary for Big Code Machine Learning (2019, background): “Ben-Nun et al. [12] define an embedding space, inst2vec, based on an Intermediate Representation of the code to recover the semantics of statements based on their context.”
A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “Existing code language models commonly include Recurrent Neural Networks (RNNs) (Jordan, 1997; Rumelhart et al., 1986; Ben-Nun et al., 2018), Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997; Iyer et al., 2016; Yin and Neubig, 2017; Alon et al., 2019b) and Transformers…”
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks (2021, methodology): “For the purpose of this comparison, we train the same MISIM model on C++1000 and GCJ-297 and test the two trained models on a third, independent dataset - POJ-104.”
Learning Execution through Neural Code Fusion (2019, methodology): “0% test accuracy, where the state-of-the-art [4] achieves 95.”