CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
By Erik Nijkamp, Bo Pang, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, Ying-Bo Zhou, Yingbo Zhou, Silvio Savarese, Cai-Ming Xiong and 1 more. 2022
1,548 citations recorded by the source corpus.
Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: https://github.com/salesforce/CodeGen.
Subfields: Information Systems, Software, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Chun Xia
- Lingming Zhang
- Wayne Xin Zhao
- Ji-rong Wen
- Baptiste Rozière
- Koushik Sen
- Philip S. Yu
- Xinyun Chen
In Software
- Chun Xia
- Lingming Zhang
- Mark Harman
- Koushik Sen
- Shin Yoo
- Armando Solar-Lezama
- Xinyun Chen
- Pushmeet Kohli
In Large Language Models
- Graham Neubig
- Jianfeng Gao
- Jie Tang
- Zhiyuan Liu
- Weizhu Chen
- Minlie Huang
- Caiming Xiong
- Edouard Grave
Citations in the corpus
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “Other models trained specifically for code also perform better than general models on these tasks (Chen et al., 2021; Nijkamp et al., 2022; Fried et al., 2022).”
- Code Llama: Open Foundation Models for Code (2023, methodology): “Code Llama 7B outperforms larger models such as CodeGen-Multi or StarCoder, and is on par with Codex (code-cushman-001, Chen et al., 2021).”
- StarCoder: may the source be with you! (2023, methodology): “We also note that CodeGen-16B-Multi appears to generate less toxic responses than StarCoderBase.”
- MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “Over the past few years, code generation models have demonstrated their utility on a wide variety of software engineering tasks, including test generation, documentation generation, and even synthesizing working programs from natural language descriptions [1], [3], [4], [5], [6].”
- Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “The C ODEX Family At the time of this work, C ODEX had three publicly available models.”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “These include CodeT5 (Wang et al., 2021, 2023), CodeGen (Nijkamp et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023b), AlphaCode (Li et al., 2022), InCoder (Fried et al., 2022), and CodeGeeX (Zheng et al., 2023).”
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, methodology): “MOSS is initialized with CodeGen(Nijkamp et al., 2022), being pretrained on 100B Chinese tokens and 20B English tokens, and has further integrated supervised fine-tuning and preference model, as well as plugin augmentation, but not all the version are publicly available.”
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, background): “5), CodeGen (Nijkamp et al., 2022) (codegen-16B-multi and codegen-16B-mono), CodeT5+ (Wang et al., 2023b) and Xgen 3 for ablation experiments.”
- CodeT: Code Generation with Generated Tests (2022, methodology): “Furthermore, we include two publicly available pre-trained models: INCODER (Fried et al., 2022a) and CODEGEN (Nijkamp et al., 2022).”
- RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation (2024, background): “…importance as code, facilitating synchronization and collaboration among teams. nAI, 2022, 2023), especially in code understanding and generation (Nijkamp et al., 2023; Li et al., 2023; Chen et al., 2021; Rozi ` ere et al., 2023; Xu et al., 2024; Sun et al., 2023; Wang et al., 2023; Khan and…”
- LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023, background): “Recently, there have been a plethora of open-source LLMs demonstrating strong capabilities in writing code, e.g., CodeGen [103], StarCoder [94], and CodeGeeX [104].”
- TheoremQA: A Theorem-driven Question Answering dataset (2023, methodology): “• MOSS-instruct-16B: This model is based on CodeGen (Nijkamp et al., 2022), which is further instruction-tuned with instruction following dataset distilled from GPT. .”
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023, background): “G EN (Nijkamp et al., 2022).”
- A Survey on Large Language Models for Code Generation (2024, neutral): “…for Optimization [40] Parameter-Efficient Fine-Tuning [41] Data Pruning for Fine-Tuning [42] B. Feedback ClarifyGPT Framework [43] RLEF [44] Crowd-Sourced RLHF [45] C. Prompt Engineering Chain-of-Thought Prompting [46] AceCoder Technique [47] Security-Focused Prompting [48] V. Evaluations”
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, methodology): “To efficiently scale up the model without the need of pretraining from scratch, we propose a compute-efficient pretraining strategy to initialize model components (i.e. encoder and decoder) of CodeT5+ with off-the-shelf pretrained LLMs [Nijkamp et al., 2023b] (see the rightmost diagram of Fig.”
- MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, methodology): “We use the multi-language parallel MultiPL-HumanEval and MultiPL-MBPP benchmarks to evaluate three state-ofthe-art code generation models: Codex [1], CodeGen [3], and InCoder [4].”
- CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context (2022, positive): “In contrast, CoCoMIC encodes the code sequence of an entity into one single to-ken, enabling the model to incorporate more cross-file context while saving the input length.”
- Communicative Agents for Software Development (2023, methodology): “(3) Software implementation is the detailed creation procedure of the software to implement the design [15; 1; 6; 28].”
- CodeS: Towards Building Open-source Language Models for Text-to-SQL (2024, neutral): “We additionally fine-tune Llama2-7B and 13B as our competitive baselines.”