code2vec: learning distributed representations of code

By Uri Alon, Meital Zilberstein, Omer Levy, Eran Yahav. 2018

1,357 citations recorded by the source corpus.

We present a neural model for representing snippets of code as continuous distributed vectors (``code embeddings''). The main idea is to represent a code snippet as a single fixed-length code vector, which can be used to predict semantic properties of the snippet. To this end, code is first decomposed to a collection of paths in its abstract syntax tree. Then, the network learns the atomic representation of each path while simultaneously learning how to aggregate a set of them. We demonstrate the effectiveness of our approach by using it to predict a method's name from the vector representation of its body. We evaluate our approach by training a model on a dataset of 12M methods. We show that code vectors trained on this dataset can predict method names from files that were unobserved during training. Furthermore, we show that our model learns useful method name vectors that capture semantic similarities, combinations, and analogies. A comparison of our approach to previous techniques over the same dataset shows an improvement of more than 75%, making it the first to successfully predict method names based on a large, cross-project corpus. Our trained model, visualizations and vector similarities are available as an interactive online demo at http://code2vec.org. The code, data and trained models are available at https://github.com/tech-srl/code2vec.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Program Synthesis with Large Language Models (2021, methodology): “…models for code have shown to be effective for transfer learning across software engineering tasks, including CuBERT [Kanade et al., 2020], CodeBERT [Feng et al., 2020], PyMT5 [Clement et al., 2020], code2vec [Alon et al., 2019], and other T5 models trained on code [Mastropaolo et al., 2021].”
You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, methodology): “They support tasks such as extracting code semantics [2, 4], and code and edit completion [3,10,21,57], with several commercial products adopting these techniques [11, 16].”
The adverse effects of code duplication in machine learning models of code (2018, background): “ics worsen, the effect differs. For example, JavaScript-150k and Java-Large have very similar (file-level) duplication but the impact of duplication on the evaluation metrics of PHOG [9] and code2vec [6] is quite different. This can be attributed to two factors (a) different models are affected diff”
Unified Pre-training for Program Understanding and Generation (2021, methodology): “While most of the prior approaches use task-specific representation learning, a few works (Alon et al., 2019b; Feng et al., 2020; Guo et al., 2021; Lachaux et al., 2020; Clement et al., 2020) attempted to learn transferable representations in an unsupervised fashion.”
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, background): “As code artiﬁcial intelligence receives more and more attention (Allamanis et al. 2015; Yin and Neubig 2017; Al-lamanis et al. 2018; Monperrus 2018; Alon et al. 2019; Svyatkovskiy et al. 2020), the evaluation of code synthesis becomes critical to promote its development.”
code2seq: Generating Sequences from Structured Representations of Code (2018, methodology): “Further, our model decodes the output sequence step-by-step while attending over the input paths, and can thus generate unseen sequences, compared to code2vec (Alon et al., 2018a) which has a closed vocabulary.”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “Following previous works (Allamanis et al., 2015; 2016; Alon et al., 2018; 2019), we also report the subtoken score, a more flexible metric which computes the precision, recall, and F1 scores for retrieving the original case-insensitive subtokens.”
Information Leakage in Embedding Models (2020, background): “Embeddings have also been successfully applied to other data domains including social networks [21], source code [2], YouTube watches [11], movie feedback [25], locations [12], etc.”
Neural Edit Completion (2020, methodology): “AST paths were shown to be an effective representation for predicting variable names, method names [Alon et al. 2019c], natural language descriptions [Alon et al. 2019a] and code completion [Alon et al. 2019b].”
PanGu-Coder: Program Synthesis with Function-Level Language Modeling (2022, background): “[4] who introduced code2vec[4], a neural model for representing snippets of code as continuous distributed vectors, extending the idea from Mikolov et al.”
CoCoGUM: Contextual Code Summarization with Multi-Relational GNN on UMLs (2020, background): “Alon et al. [53, 54] represent a code snippet as a set of compositional paths in the abstract syntax trees.”
Learning to Prove Theorems via Interacting with Proof Assistants (2019, background): “A key ingredient is using a deep network to embed a program into a vector, by treating the program as a sequence of tokens (Allamanis et al., 2016b) or using structured input such as ASTs (Allamanis et al., 2016a; Alon et al., 2018) or graphs (Allamanis et al., 2017).”
Learning-Based Recursive Aggregation of Abstract Syntax Trees for Code Clone Detection (2019, methodology): “Researchers have also investigated the semantics captured in these embeddings, either by vector arithmetic yielding word analogies [19], [20] (as in the follow-up paper to word2vec method [27]), or by plotting a neighbourhood-preserving 2D projection [23].”
Learning to Spot and Refactor Inconsistent Method Names (2019, methodology): “Thus, we leverage only static and syntactic information in our approach and rely on deep learning representations that have been shown to be effective capturing semantics even for code [54], [73].”
When deep learning met code search (2019, methodology): “Existing work has introduced neural networks to identify software defects [10], guide program synthesis [5, 35], enable new representations for program analysis [1, 3], facilitate code reuse [17], and automate code changes [34].”
COSET: A Benchmark for Evaluating Neural Program Embeddings (2019, neutral): “A number of new deep-learning architectures developed to specifically address this issue [2; 3; 4; 11; 12] tried a variety”
Structural Language Models of Code (2019, methodology): “Whereas prior work used whole paths (between two leaf nodes) to encode ASTs (Alon et al., 2019a;b), our model observes partial paths (between a leaf and any other node) and learns to extend them by predicting their next node.”
Adversarial examples for models of code (2019, methodology): “To evaluate the performance of the attack and the defense, we used code2vec [6] as a downstream model.”