CodeT: Code Generation with Generated Tests
By Fengji Zhang, Anh Gia-Tuan Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, Weizhu Chen. 2022
560 citations recorded by the source corpus.
The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samples. However, a major challenge for this task is to select the most appropriate solution from the multiple samples generated by the pre-trained language models. A natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases, but the manual creation of such test cases is often costly and time-consuming. In this paper, we propose a novel method, CodeT, that leverages the same pre-trained language models to automatically generate test cases for the code samples, thus reducing the human effort and increasing the coverage of the test scenarios. CodeT then executes the code samples using the generated test cases, and performs a dual execution agreement, which considers both the consistency of the outputs against the generated test cases and the agreement of the outputs with other code samples. We conduct comprehensive experiments on four benchmarks, HumanEval, MBPP, APPS and CodeContests, using five different pre-trained language models with varying sizes and capabilities. Our results show that CodeT can significantly improve the performance of code solution selection over previous methods, achieving remarkable and consistent gains across different models and benchmarks. For instance, CodeT improves the pass@1 metric on HumanEval to 65.8%, which represents an absolute improvement of 18.8% over the code-davinci-002 model, and an absolute improvement of more than 20% over the previous state-of-the-art results.
Subfields: Computer Networks and Communications, Information Systems, Software, Natural Language Processing, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Joseph E. Gonzalez
- Baptiste Rozière
- Ion Stoica
- Armando Solar-Lezama
- Ge Li
- Xinyun Chen
- Xi Victoria Lin
- Shin Yoo
In Information Systems
- Xi Victoria Lin
- Wen-tau Yih
- Koushik Sen
- Graham Neubig
- Jianfeng Gao
- Baptiste Rozière
- Xinyun Chen
- Mark Harman
In Software
- Mark Harman
- Koushik Sen
- Shin Yoo
- Xi Victoria Lin
- Josh Tenenbaum
- Denny Zhou
- Armando Solar-Lezama
- Gabriel Synnaeve
In Natural Language Processing
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Mohit Bansal
- Caiming Xiong
- Wen-tau Yih
- Richard Socher
In Code Generation & Program Synthesis
Citations in the corpus
- Code Llama: Open Foundation Models for Code (2023, methodology): “However, instead of downscaling frequencies linearly as Chen et al. (2023b), we change the base period from which they are derived.”
- MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “2) Python Results and Replication: Our Python MBPP pass@1 rates for Codex are slightly higher than what is re-ported in [34] (60.3% compared to 58.1%).”
- Execution-Based Evaluation for Open-Domain Code Generation (2022, background): “Some propose to leverage the power of neural LMs (Tufano et al., 2020; Li et al., 2022), even jointly considering so-lution and test case generation (Chen et al., 2022).”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Furthermore, Chen et al. (2022) demonstrated that LLMs can assist in generating test case inputs/outputs for competitive programming problems, thereby improving the accuracy of the generated code, thus inspiring our test generation scenario.”
- Reflexion: language agents with verbal reinforcement learning (2023, unknown): “CodeT [4] uses self-generated unit tests that are used to score generated function implementations.”
- LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, methodology): “Chen et al. (2022a) and other works (Tu-fano et al., 2020; Li et al., 2022a) use LMs to generate test cases instead of directly judging the correctness of the output programs.”
- Self-Edit: Fault-Aware Code Editor for Code Generation (2023, methodology): “Recently some approaches are also proposed to post-process programs generated by LLMs (Shi et al., 2022; Inala et al., 2022; Chen et al., 2022; Zhang et al., 2022).”
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, methodology): “Chen et al. (2022a); Zhang et al. (2023b) proposed C ODE T and ALGO , that use generated tests (using LLM or brute-force solution) to re-rank the generated solutions.”
- MetaGPT: Meta Programming for Multi-Agent Collaborative Framework (2023, methodology): “We conduct a comparative evaluation by comparing recent domain-specific LLMs in the code generation field, including AlphaCode [28], Incoder [43], CodeGeeX [44], and CodeX [24] and CodeT [45], with general domain LLMs such as PaLM [46], GPT-4 [47], and MetaGPT.”
- A Survey on Large Language Models for Code Generation (2024, unknown): “2 Overview: LLMs for Code Generations”
- Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, methodology): “We follow Chen et al. (2023a) and use an LM to generate a synthetic test suite of syntactically valid “assert” statements for each question.”
- CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024, methodology): “When occurring errors, this tool will offer error information to facilitate LLMs to fix bugs until programs are error-free, which has been proven to be effective by many existing works (Chen et al., 2022; Zhang et al., 2023b) to improve the accuracy of output programs.”
- FacTool: Factuality Detection in Generative AI - A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios (2023, methodology): “For response annotation, we annotate ChatGPT’s responses using the execution code provided in (Chen et al., 2022) against the HumanEval test”
- Planning with Large Language Models for Code Generation (2023, methodology): “We follow the prompt design in Chen et al. (2022) to automatically generate test cases and run our PG-TD algorithm using the automatically-generated test cases.”
- Demystifying GPT Self-Repair for Code Generation (2023, methodology): “For example , let the initial positions be $x = [1, 2, 4, 4]$.”
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, background): “, 2023]; and (iii) models with enhancement generation strategies (bottom) such as unit test generation [Chen et al., 2023] and prompting.”
- Structured Chain-of-Thought Prompting for Code Generation (2023, unknown): “We can see that the performance of ChatGPT is continually improved by adding CodeT and SCoT prompting.”
- Binding Language Models in Symbolic Languages (2022, methodology): “We base our experiments for B INDER on Codex, a code-pretrained version of GPT-3, which has been shown to perform proficiently on code generation tasks (Rajkumar et al., 2022; Chen et al., 2022) and even some weakly-related tasks like dialogue state tracking (Hu et al., 2022).”