IntelliCode compose: code generation using transformer

By Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan. 2020

626 citations recorded by the source corpus.

In software development through integrated development environments (IDEs), code completion is one of the most widely used features. Nevertheless, majority of integrated development environments only support completion of methods and APIs, or arguments. In this paper, we introduce IntelliCode Compose – a general-purpose multilingual code completion tool which is capable of predicting sequences of code tokens of arbitrary types, generating up to entire lines of syntactically correct code. It leverages state-of-the-art generative transformer model trained on 1.2 billion lines of source code in Python, C#, JavaScript and TypeScript programming languages. IntelliCode Compose is deployed as a cloud-based web service. It makes use of client-side tree-based caching, efficient parallel implementation of the beam search decoder, and compute graph optimizations to meet edit-time completion suggestion requirements in the Visual Studio Code IDE and Azure Notebook. Our best model yields an average edit similarity of 86.7% and a perplexity of 1.82 for Python programming language.

Subfields: Artificial Intelligence, Information Systems, Software, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Prompting & Reasoning

Citations in the corpus

Competition-level code generation with AlphaCode (2022, background): “Lanza, 2008) and large amounts of existing code data (Aye et al., 2021; Svyatkovskiy et al., 2020).”
Program Synthesis with Large Language Models (2021, background): “Code completion has been a particular focus of interest [Raychev et al., 2016, Karampatsis et al., 2020, Svyatkovskiy et al., 2020, Kim et al., 2020].”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, background): “From LISPbased pioneering deductive synthesis approaches [31, 37] tomodern program synthesis systems [23, 30], to end-to-end code generation via deep neural networks [19, 32, 33], tremendous efforts have been made to enable machines to automatically write correct programs as part of the quest to art”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “CuBERT employs BERT’s powerful masked language modeling objective to derive generic code-speciﬁc representation, and CodeBERT further adds a replaced token detection (Clark et al., 2020) task to learn NL-PL cross-modal representation.”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “Similarly, developers who are confused about what to write next can use code completion systems [57, 1, 56, 66, 67, 8, 27, 7] to automatically complete the following tokens based on the edits made to the code.”
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, background): “As code artiﬁcial intelligence receives more and more attention (Allamanis et al. 2015; Yin and Neubig 2017; Al-lamanis et al. 2018; Monperrus 2018; Alon et al. 2019; Svyatkovskiy et al. 2020), the evaluation of code synthesis becomes critical to promote its development.”
PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers (2020, methodology): “…in software development environments: language modeling or generation can be used for code completion (Raychev et al., 2014; Bruch et al., 2009; Svyatkovskiy et al., 2019, 2020), translation/summarization to generate documentation or natural language summaries (Moreno et al., 2013; Scalabrino…”
GraphCodeBERT: Pre-training Code Representations with Data Flow (2020, background): “Existing works (Kanade et al., 2019; Karampatsis & Sutton, 2020; Feng et al., 2020; Svyatkovskiy et al., 2020; Buratti et al., 2020) regard a source code as a sequence of tokens and pre-train models on source code to support code-related tasks such as code search, code completion, code summarization”
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set? (2022, methodology): “This has been achieved, for example, by (i) exploiting contextual information such as the code surrounding the statements on which code completion is triggered [8, 48] and/or or the recent code change history [48]; and (ii) moving towards more advanced techniques such as Deep Learning (DL) models, t”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X (2023, methodology): “From LISP-based pioneering deductive synthesis approaches (Waldinger and Lee, 1969; Summers, 1977) to modern program synthesis systems (Solar-Lezama, 2008; Polozov and Gulwani, 2015), to end-to-end code generation via deep neural networks (Mou et al., 2015; Svyatkovskiy et al., 2020; Sun et al., 202”
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2023, background): “As mentioned in Section 2, we especially focus on the code generated with Python3 language, since it is one of the most widely studied programming languages in code generation [4, 12, 17, 36, 59, 61, 64].”
Productivity assessment of neural code completion (2022, methodology): “The IntelliCode Compose system uses the term CTR (Click Through Rate) for this and reports a value of 10% in online trials [12].”
LongCoder: A Long-Range Pre-trained Language Model for Code Completion (2023, methodology): “We follow Lu et al. (2021) to evaluate the performance of the models in terms of Exact Match (EM) and Edit Similarity (Edit Sim) on a per-line basis (Svyatkovskiy et al., 2020).”
CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, methodology): “We pit CodeFill against four baseline models and two the-stateof-the-art models, namely GPT-C [49] and TravTrans+ [25].”
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, neutral): “Edit Sim (Lev-enshtein distance) is popularly used in code generation evaluation (Svyatkovskiy et al., 2020).”
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, methodology): “…such as [Wang et al., 2021b] pretrained with span denoising objective are not ideal for auto-regressive generation tasks like next-line code completion [Lu et al., 2021, Svyatkovskiy et al., 2020b], as these models are trained to recover short spans of limited lengths rather than a whole program.”
Energy-Based Models for Code Generation under Compilability Constraints (2021, background): “…on perplexity and similarity with respect to ground truth completions (in terms of exact-match accuracy, Levens-thein distance and ROUGE scores) (Svyatkovskiy et al., 2020a; Lu et al., 2021), usually failing to measure and control for compilability of generated sequences or semantic and…”
PlotCoder: Hierarchical Decoding for Synthesizing Visualization Code in Programmatic Context (2021, methodology): “Our code synthesis problem is also related to code completion, i.e., autocompleting the program given the code context (Raychev et al., 2014; Li et al., 2018; Svyatkovskiy et al., 2020).”
Programming Puzzles (2021, background): “[54] Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan.”