code2seq: Generating Sequences from Structured Representations of Code

By Uri Alon, Shaked Brody, Omer Levy, Eran Yahav. 2018

812 citations recorded by the source corpus.

The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval. Sequence-to-sequence (seq2seq) models, adopted from neural machine translation (NMT), have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens. We present ${\rm {\scriptsize CODE2SEQ}}$: an alternative approach that leverages the syntactic structure of programming languages to better encode source code. Our model represents a code snippet as the set of compositional paths in its abstract syntax tree (AST) and uses attention to select the relevant paths while decoding. We demonstrate the effectiveness of our approach for two tasks, two programming languages, and four datasets of up to $16$M examples. Our model significantly outperforms previous models that were specifically designed for programming languages, as well as state-of-the-art NMT models. An interactive online demo of our model is available at this http URL Our code, data and trained models are available at this http URL

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, background): “Code2seq (Alon et al., 2018) found that ASTs could also be leveraged for code-to-text generation.”
You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, methodology): “They support tasks such as extracting code semantics [2, 4], and code and edit completion [3,10,21,57], with several commercial products adopting these techniques [11, 16].”
Unified Pre-training for Program Understanding and Generation (2021, methodology): “Code Summarization (MovshovitzAttias and Cohen, 2013; Allamanis et al., 2016; Iyer et al., 2016; Alon et al., 2019a; Hu et al., 2018; Harer et al., 2019; Ahmad et al., 2020), Bug Detection (Ray et al.”
CodeBERT: A Pre-Trained Model for Programming and Natural Languages (2020, neutral): “However, our model achieve slightly lower results than code2seq (Alon et al., 2019).”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, background): “[4] consider the task of summarizing functions to their names.”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “…Wei et al., 2023a; Malik et al., 2019), code summarization (LeClair et al., 2019; Iyer et al., 2016; Barone and Sennrich, 2017; Hasan et al., 2021; Alon et al., 2018), code security (Liguori et al., 2022; Pearce et al., 2022; Tony et al., 2023), etc. 2023) demonstrated contamination by simply…”
PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers (2020, background): “…translation/summarization to generate documentation or natural language summaries (Moreno et al., 2013; Scalabrino et al., 2017; Wan et al., 2018; Alon et al., 2018) or even summarize a set of code changes (Moreno et al., 2014), translation and grammar error correction to patch and detect bugs…”
Unsupervised Translation of Programming Languages (2020, background): “To address this issue, several approaches proposed to use additional constraints on the decoder, to ensure that the generated functions respect the syntax of the target language [3, 4, 5, 40, 48].”
IntelliCode compose: code generation using transformer (2020, background): “Some of the early applications of machine learning of source code include code search [1, 2], bug detection and localization [3], program synthesis [4], code summarization [5] and code completion [6–10].”
GraphCodeBERT: Pre-training Code Representations with Data Flow (2020, background): “, 2020), code summarization (Alon et al., 2018; Hu et al., 2018) and so on (Nguyen & Nguyen, 2015; Allamanis et al.”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “[8] only keeps the 190,000 most common vocabulary words.”
Structural Language Models for Any-Code Generation (2019, methodology): “Whereas prior work used whole paths (between two leaf nodes) to encode an AST (Alon et al., 2019b;a), our model observes partial paths (between a leaf and an intermediate node) and learns to extend them.”
Open Graph Benchmark: Datasets for Machine Learning on Graphs (2020, methodology): “[6, 7], we use an F1 score to evaluate predicted sub-tokens against ground-truth sub-tokens.”
Global Relational Models of Source Code (2020, background): “, 2018) and code2seq (Alon et al., 2019) model programs as a weighted combination of a set of leaf-to-leaf paths in the abstract syntax tree.”
Structured Neural Summarization (2018, methodology): “We use the datasets and splits of Alon et al. (2019) provided by their website.”
Backdoors in Neural Models of Source Code (2020, methodology): “What learned representations can be used in the domain of source code? We consider two different model architectures: the first simply treats code as a sequence of tokens (like LSTMs); the second takes the abstract syntax tree (AST) of the code as input [13].”