Program Synthesis with Large Language Models
By Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le and 1 more. 2021
4,124 citations recorded by the source corpus.
This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6 percent of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8 percent accuracy. Going further, we study the model's ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model's initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.
Subfields: Artificial Intelligence, Information Systems, Software, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Quoc V. Le
- Graham Neubig
- Yoshua Bengio
- Yejin Choi
In Information Systems
- Percy Liang
- Charles Sutton
- Uri Alon
- Miltiadis Allamanis
- Graham Neubig
- Wen-tau Yih
- Xinyun Chen
- Chun Xia
In Software
- Josh Tenenbaum
- Rishabh Singh
- Armando Solar-Lezama
- Oleksandr Polozov
- Percy Liang
- Pushmeet Kohli
- Charles Sutton
- Chun Xia
In Prompting & Reasoning
- Denny Zhou
- Luke S. Zettlemoyer
- Wenhu Chen
- Xinyun Chen
- Sean Welleck
- Jason Wei
- Percy Liang
- Graham Neubig
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “See Austin et al. (2021) for a similar experiment with the MathQA dataset (Amini et al., 2019).”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “Params HumanEval MBPP pass@ @1 @100 @1 @80 LaMDA 137B 14.0 47.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We report the average pass@1 scores of our models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021).”
- Mistral 7B (2023, neutral): “…(0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated results: MMLU [12] (5-shot), BBH [23] (3-shot), and AGI Eval [29] (3-5-shot, English multiple-choice questions only)…”
- Mixtral of Experts (2024, neutral): “…(0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated results: MMLU [16] (5-shot), BBH [29] (3-shot), and AGI Eval [34] (3-5-shot, English multiple-choice questions only)…”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “, 2020), as well as the use of large language models (Chen et al., 2021; Austin et al., 2021).”
- Solving Quantitative Reasoning Problems with Language Models (2022, background): “Previous research has shown that large language models achieve impressive performance on math and programming questions after training on domain specific datasets (Chen et al., 2021; Austin et al., 2021; Drori et al., 2021).”
- Competition-level code generation with AlphaCode (2022, methodology): “Similar to our sampling and filtering, though on a much smaller scale, Chen et al. (2021), Austin et al. (2021), and Cobbe et al. (2021) found that repeated sampling on the same problem significantly increases the probability of finding a correct solution.”
- Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, positive): “Our evaluation dataset is the Mostly Basic Python Programming Problems (MBPP) dataset, introduced in Austin et al. (2021).”
- Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “For models of O(100B) parameters, recent work has finetuned task-specific models for program synthesis (Austin et al., 2021; Chen et al., 2021), summarization (Wu et al., 2021), as well as improved bias and fairness behavior (Solaiman & Dennison, 2021).”
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “We compare model pass@1 scores on the HumanEval [18] and MBPP [10] left-to-right synthesis benchmarks, with results in Table 7.”
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, methodology): “Several recent and concurrent efforts explore using large language models for program synthesis (Chen et al., 2021; Austin et al., 2021; Li et al., 2022; Fried et al., 2022) and its effectiveness (Vaithilingam et al.”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “The range of feasible program spaces to search over has also grown, from historically restrictive domain-specific languages to general-purpose programming languages (Austin et al., 2021).”
- A General Language Assistant as a Laboratory for Alignment (2021, background): “The recent work [AON21] evaluated language models (without a great deal of code training) on code, including in a conversational manner.”
- Code Llama: Open Foundation Models for Code (2023, methodology): “We start by reporting results for Python code generation using the HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021) and APPS (Hendrycks et al., 2021) benchmarks.”
- Training Language Models with Language Feedback (2022, background): “Other work uses language feedback at test time to correct mistakes in a model’s behavior, for e.g. image segmentation (Rupprecht et al., 2018) or code generation (Elgohary et al., 2020; Austin et al., 2021).”
- Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions (2021, neutral): “Although prior research has evaluated the functionality of code generated by language models [3], [2], there is no B. Dolan-Gavitt is supported in part by the National Science Foundation award #1801495.”