CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X

By Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shanshan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li and 3 more. 2023

573 citations recorded by the source corpus.

Large pre-trained code generation models, such as OpenAI Codex, can generate syntax-and function-correct code, making the coding of programmers more productive. In this paper, we introduce CodeGeeX, a multilingual model with 13 billion parameters for code generation. CodeGeeX is pre-trained on 850 billion tokens of 23 programming languages as of June 2022. Our extensive experiments suggest that CodeGeeX outperforms multilingual code models of similar scale for both the tasks of code generation and translation on HumanEval-X. Building upon HumanEval (Python only), we develop the HumanEval-X benchmark for evaluating multilingual models by hand-writing the solutions in C++, Java, JavaScript, and Go. In addition, we build CodeGeeX-based extensions on Visual Studio Code, JetBrains, and Cloud Studio, generating 8 billion tokens for tens of thousands of active users per week. Our user study demonstrates that CodeGeeX can help to increase coding efficiency for 83.4% of its users. Finally, CodeGeeX is publicly accessible since Sep. 2022, we open-sourced its code, model weights, API, extensions, and HumanEval-X at https://github.com/THUDM/CodeGeeX.

Subfields: Information Systems, Software, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

  1. Chun Xia
  2. Lingming Zhang
  3. Jiawei Liu
  4. Yuyao Wang
  5. Denny Zhou
  6. Ming-Ho Yee
  7. Armando Solar-Lezama
  8. Xiao Liu

In Software

  1. Chun Xia
  2. Lingming Zhang
  3. Armando Solar-Lezama
  4. Jiawei Liu
  5. Yuyao Wang
  6. Xinyun Chen
  7. Ming-Ho Yee
  8. Denny Zhou

In Evaluation & Benchmarks

  1. Chun Xia
  2. Lingming Zhang
  3. Armando Solar-Lezama
  4. Qian Liu
  5. Xinyun Chen
  6. Fei Huang
  7. Chang Zhou
  8. Siva Reddy

In Code Generation & Program Synthesis

  1. Armando Solar-Lezama
  2. Qian Liu
  3. Xinyun Chen
  4. Raymond Li
  5. Carolyn Jane Anderson
  6. Chun Xia
  7. Koushik Sen
  8. Maxwell Lin

Citations in the corpus