CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

By Erik Nijkamp, Bo Pang, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, Ying-Bo Zhou, Yingbo Zhou, Silvio Savarese, Cai-Ming Xiong and 1 more. 2022

1,548 citations recorded by the source corpus.

Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: https://github.com/salesforce/CodeGen.

Subfields: Information Systems, Software, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

  1. Chun Xia
  2. Lingming Zhang
  3. Wayne Xin Zhao
  4. Ji-rong Wen
  5. Baptiste Rozière
  6. Koushik Sen
  7. Philip S. Yu
  8. Xinyun Chen

In Software

  1. Chun Xia
  2. Lingming Zhang
  3. Mark Harman
  4. Koushik Sen
  5. Shin Yoo
  6. Armando Solar-Lezama
  7. Xinyun Chen
  8. Pushmeet Kohli

In Large Language Models

  1. Graham Neubig
  2. Jianfeng Gao
  3. Jie Tang
  4. Zhiyuan Liu
  5. Weizhu Chen
  6. Minlie Huang
  7. Caiming Xiong
  8. Edouard Grave

Citations in the corpus