Competition-level code generation with AlphaCode
By Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom, Eccles, James Keeling, Felix Gimeno and 22 more. 2022
2,356 citations recorded by the source corpus.
Programming is a powerful and ubiquitous problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent transformer-based neural network models show impressive code generation abilities yet still perform poorly on more complex tasks requiring problem-solving skills, such as competitive programming problems. Here, we introduce AlphaCode, a system for code generation that achieved an average ranking in the top 54.3% in simulated evaluations on recent programming competitions on the Codeforces platform. AlphaCode solves problems by generating millions of diverse programs using specially trained transformer-based networks and then filtering and clustering those programs to a maximum of just 10 submissions. This result marks the first time an artificial intelligence system has performed competitively in programming competitions. Description Machine learning systems can program too Computer programming competitions are popular tests among programmers that require critical thinking informed by experience and creating solutions to unforeseen problems, both of which are key aspects of human intelligence but challenging to mimic by machine learning models. Using self-supervised learning and an encoder-decoder transformer architecture, Li et al. developed AlphaCode, a deep-learning model that can achieve approximately human-level performance on the Codeforces platform, which regularly hosts these competitions and attracts numerous participants worldwide (see the Perspective by Kolter). The development of such coding platforms could have a huge impact on programmers’ productivity. It may even change the culture of programming by shifting human work to formulating problems, with machine learning being the main one responsible for generating and executing codes. —YS Modern machine learning systems can achieve average human-level performance in popular competitive programming contests.
Subfields: Artificial Intelligence, Information Systems, Software, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Graham Neubig
- Yoshua Bengio
- Yejin Choi
In Information Systems
- Percy Liang
- Uri Alon
- Wen-tau Yih
- Xinyun Chen
- Luke S. Zettlemoyer
- Graham Neubig
- Denny Zhou
- Rishabh Singh
In Software
- Armando Solar-Lezama
- Rishabh Singh
- Oleksandr Polozov
- Josh Tenenbaum
- Percy Liang
- Pushmeet Kohli
- Koushik Sen
- Michihiro Yasunaga
In Code Generation & Program Synthesis
- Charles Sutton
- Paul Denny
- Armando Solar-Lezama
- Juho Leinonen
- Graham Neubig
- Sami Sarsa
- Arto Hellas
- Oleksandr Polozov
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, background): “Recent work has shown that large language models can be useful for coding tasks including competitive programming (Li et al., 2022), code completion (Chen et al., 2021), and program synthesis from natural language specifications (Chen et al., 2021; Austin et al., 2021).”
- Solving Quantitative Reasoning Problems with Language Models (2022, background): “Our focus on pass@1 and majority voting as the primary evaluation metrics is due in part to the fact that they are less susceptible to false positives than pass@k [17].”
- Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, neutral): “…language models exhibit surprisingly impressive capabilities (Devlin et al., 2019; Brown et al., 2020), including the ability to generate code that solves simple programming problems (Chen et al., 2021; Austin et al., 2021) and problems from programming competitions (Li et al., 2022).”
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, methodology): “Our Contributions Our work shares the basic idea of adopting language models for program synthesis with the recent and concurrent efforts (Chen et al., 2021; Austin et al., 2021; Li et al., 2022) with a single-turn user intent specification.”
- Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “We note that a similar score estimation scheme has been used in previous works (Li et al., 2022; Maclaurin & Adams, 2015).”
- Scaling Instruction-Finetuned Language Models (2022, background): “, 2022) and program synthesis data (Yasunaga and Liang, 2020; Li et al., 2022).”
- Code Llama: Open Foundation Models for Code (2023, methodology): “…& Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022; Chen et al., 2023a), as well as solving math problems as demonstrated by PaLM (Chowdhery et al., 2022) or Codex (Chen et al., 2021).”
- MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “AlphaCode and PaLM-Coder are not available for academic researchers to investigate.”
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, neutral): “First, different from Codex in [7], both CodeGeeX—the model itself—and how such scale of code models can be pre-trained are open-sourced, facilitating the understanding and advances in pre-trained code models.”
- Foundation Models and Fair Use (2023, background): “Li et al. (2022) filter their dataset of source code collected from GitHub by license to train AlphaCode.”
- Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “These findings suggest that common practices to improve LLMs may not address the complexities inherent in open-domain code generation problems.”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Methods such as AlphaCode (Li et al., 2022), AlphaCode 2(Gemini Team et al., 2023), ALGO (Zhang et al., 2023d), Parsel (Zelikman et al., 2022), code cleaning (Jain et al., 2023), code explainations (Li et al., 2023a), analogical reasoning (Yasunaga et al., 2023), and AlphaCodium (Ridnik et al.,…”