A systematic evaluation of large language models of code

By Frank F. Xu, Uri Alon, Graham Neubig, Vincent J. Hellendoorn. 2022

884 citations recorded by the source corpus.

Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing opensource models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, that was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area. We have an online appendix at https://arxiv.org/abs/2202.13169.

Subfields: Artificial Intelligence, Information Systems, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “Large language models trained on vast repositories of code have demonstrated remarkable progress in neural program synthesis and related tasks [18, 10, 68, 46, 21].”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “…made big splashes as generative models in areas as diverse as text-to-image synthesis (Ramesh et al., 2022; Crowson et al., 2022; Rombach et al., 2022), protein modeling (Jumper et al., 2021; Ahdritz et al., 2022), and computer programming (Chen et al., 2021; Xu et al., 2022; Fried et al., 2022).”
StarCoder: may the source be with you! (2023, background): “Most of these models have not been fully open, but PolyCoder (Xu et al., 2022) and SantaCoder (Ben Allal et al., 2023) are notable exceptions and have both open models and training data.”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “Over the past few years, code generation models have demonstrated their utility on a wide variety of software engineering tasks, including test generation, documentation generation, and even synthesizing working programs from natural language descriptions [1], [3], [4], [5], [6].”
SantaCoder: don't reach for the stars! (2023, neutral): “…token sequence) CodeGen (Nijkamp et al., 2022) Exact Match (sha-256) AlphaCode (Li et al., 2022) Exact Match (non-whitespace text) PolyCoder (Xu et al., 2022a) Exact Match (hash) PaLM Coder (Chowdhery et al., 2022) Near-deduplication (Levenshtein distance) CodeParrot (Tunstall et al., 2022)…”
The Stack: 3 TB of permissively licensed source code (2022, methodology): “Several groups have explored decoder-only models with a causal language modeling objective (Chen et al., 2021; Austin et al., 2021; Nijkamp et al., 2022; Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022) and generally found that larger models are increasingly capable of synthesizing p”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “Many other recent works corroborated this finding (Nijkamp et al., 2022; Fried et al., 2022; Xu et al., 2022; Black et al., 2022), and additional techniques at inference time further improve the performance (Poesia et al.”
CodeT: Code Generation with Generated Tests (2022, methodology): “Other open-source code generation models include GPT-Neo (Black et al., 2021), GPT-J (Wang & Komatsuzaki, 2021), CodeParrot (Tunstall et al., 2022), PolyCoder (Xu et al., 2022), C ODE G EN (Nijkamp et al., 2022), and I N C ODER (Fried et al., 2022a).”
Self-Edit: Fault-Aware Code Editor for Code Generation (2023, methodology): “, 2020), PolyCoder (Xu et al., 2022), CodeGen (Nijkamp et al.”
Can Foundation Models Wrangle Your Data? (2022, methodology): “Some examples include code generation [84], Trivia QA [18, 48] and common sense reasoning tasks [18].”
Fuzz4ALL: Universal Fuzzing with Large Language Models (2023, background): “Recent developments in natural language processing (NLP) has lead to the wide-spread adoption of large language models (LLMs) for both natural language [8] and code tasks [80].”
Evaluating Large Language Models in Class-Level Code Generation (2024, background): “To date, researchers have proposed various LLMs [10, 13, 19, 27, 30, 42, 49, 53, 65, 75, 76] (such as GPT-4 [53], WizardCoder [49], and Instruct-CodeGen [10]) by training large models with over billions of parameters on massive general or code-specific corpora and instructions.”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “On the other hand, programming data drawn from online question-answering platforms like Stack Exchange [54], along with public software repositories such as GitHub, provide raw material rich with code snippets, comments, and documentation.”
Generative artificial intelligence in supply chain and operations management: a capability-based framework for analysis and implementation (2024, neutral): “Moreover, the Interaction capability becomes apparent as the GPT-3 Codex serves as an intelligent intermediary between human inputs and the resultant simulations.”
Teaching Large Language Models to Self-Debug (2023, background): “In particular, recent large language models have demonstrated a significant leap in improvement over prior deep neural networks (Chen et al., 2021a; Nijkamp et al., 2023; Zheng et al., 2023; Xu et al., 2022).”
Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2022, background): “While COMFORT has demonstrated the potential of LLMs for fuzzing, it did not leverage state-of-the-art LLMs for code and thus requires an extensive fine-tuning dataset.”
A Survey of Large Language Models (2023, neutral): “The first source is from programming question answering communities like Stack Exchange [134, 135].”