Code Prediction by Feeding Trees to Transformers

By Seohyun Kim, Jinman Zhao, Yuchi Tian, Satish Chandra. 2020

241 citations recorded by the source corpus.

Code prediction, more specifically autocomplete, has become an essential feature in modern IDEs. Autocomplete is more effective when the desired next token is at (or close to) the top of the list of potential completions offered by the IDE at cursor position. This is where the strength of the underlying machine learning system that produces a ranked order of potential completions comes into play. We advance the state-of-the-art in the accuracy of code prediction (next token prediction) used in autocomplete systems. Our work uses Transformers as the base neural architecture. We show that by making the Transformer architecture aware of the syntactic structure of code, we increase the margin by which a Transformer-based system outperforms previous systems. With this, it outperforms the accuracy of several state-of-the-art next token prediction systems by margins ranging from 14% to 18%. We present in the paper several ways of communicating the code structure to the Transformer, which is fundamentally built for processing sequence data. We provide a comprehensive experimental evaluation of our proposal, along with alternative design choices, on a standard Python dataset, as well as on Facebook internal Python corpus. Our code and data preparation pipeline will be available in open source.

Subfields: Artificial Intelligence, Information Systems, Software, Natural Language Processing, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Natural Language Processing

In Code Generation & Program Synthesis

Citations in the corpus

Program Synthesis with Large Language Models (2021, background): “Code completion has been a particular focus of interest [Raychev et al., 2016, Karampatsis et al., 2020, Svyatkovskiy et al., 2020, Kim et al., 2020].”
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “Machine learning is frequently used on tasks involving programming languages, including code completion [28, 33, 35, 42], bug detection and code repair [15, 38, 46], code summarization [8, 25], clone detection [2, 47, 48], code search [13, 23] and code translation [14, 41].”
GraphCodeBERT: Pre-training Code Representations with Data Flow (2020, background): “Neural Networks with Code Structure In recent years, some neural networks leveraging code structure such as AST have been proposed and achieved strong performance in code-related tasks like code completion (Li et al., 2017; Alon et al., 2019; Kim et al., 2020), code generation (Rabinovich et al.”
Integrating Tree Path in Transformer for Code Representation (2021, methodology): “Actually, lots of works have introduced these paths into many novel models [Kim et al., 2020, Alon et al., 2018, 2020].”
Learning Autocompletion from Real-World Datasets (2020, methodology): “Software language models are typically trained and evaluated offline on large corpora of existing source code [2]– [5], [7], [8].”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, methodology): “Code completion is nowadays a “killer” feature for Integrated Development Environments (IDEs) [7, 35, 48], and it can substantially”
Multi-task Learning based Pre-trained Language Model for Code Completion (2020, methodology): “[26] presented a transformer model for code prediction and incorporated syntactic structure into the transformer to further improve the model’s performance.”
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models (2022, background): “[21] developed a transformer architecture that is aware of code structures using abstract syntax trees.”
Learning to Generate Code Sketches (2021, background): “Similar to recent works [38, 5, 16], G RAMMFORMER loosens the CFG assumptions but retains many aspects, discussed next.”
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation (2023, methodology): “Neural techniques are rapidly being adopted for solving various Software Engineering problems, with promising results in several domains including code completion [34], [35], [36], [37], [38], program repair [39], [40], [41], and bug-ﬁnding [42], [43].”
CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, methodology): “We pit CodeFill against four baseline models and two the-stateof-the-art models, namely GPT-C [49] and TravTrans+ [25].”