CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
By Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shanshan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li and 3 more. 2023
573 citations recorded by the source corpus.
Large pre-trained code generation models, such as OpenAI Codex, can generate syntax-and function-correct code, making the coding of programmers more productive. In this paper, we introduce CodeGeeX, a multilingual model with 13 billion parameters for code generation. CodeGeeX is pre-trained on 850 billion tokens of 23 programming languages as of June 2022. Our extensive experiments suggest that CodeGeeX outperforms multilingual code models of similar scale for both the tasks of code generation and translation on HumanEval-X. Building upon HumanEval (Python only), we develop the HumanEval-X benchmark for evaluating multilingual models by hand-writing the solutions in C++, Java, JavaScript, and Go. In addition, we build CodeGeeX-based extensions on Visual Studio Code, JetBrains, and Cloud Studio, generating 8 billion tokens for tens of thousands of active users per week. Our user study demonstrates that CodeGeeX can help to increase coding efficiency for 83.4% of its users. Finally, CodeGeeX is publicly accessible since Sep. 2022, we open-sourced its code, model weights, API, extensions, and HumanEval-X at https://github.com/THUDM/CodeGeeX.
Subfields: Information Systems, Software, Evaluation & Benchmarks, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
In Software
- Chun Xia
- Lingming Zhang
- Armando Solar-Lezama
- Jiawei Liu
- Yuyao Wang
- Xinyun Chen
- Ming-Ho Yee
- Denny Zhou
In Evaluation & Benchmarks
In Code Generation & Program Synthesis
- Armando Solar-Lezama
- Qian Liu
- Xinyun Chen
- Raymond Li
- Carolyn Jane Anderson
- Chun Xia
- Koushik Sen
- Maxwell Lin
Citations in the corpus
- StarCoder: may the source be with you! (2023, methodology): “Decoder-only Transformer architectures have produced strong generative models of code, typically by training on mixtures of text and code from GitHub (Chen et al., 2021; Austin et al., 2021; Fried et al., 2022; Zheng et al., 2023; Nijkamp et al., 2023).”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “These include CodeT5 (Wang et al., 2021, 2023), CodeGen (Nijkamp et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023b), AlphaCode (Li et al., 2022), InCoder (Fried et al., 2022), and CodeGeeX (Zheng et al., 2023).”
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2023, methodology): “Although using LLM as examiner may reduce this problem (Bai et al., 2023; An et al., 2023), the runtime overhead for evaluation may be high, and LLM also has bias when used as an evaluation metric (Zheng et al., 2023a).”
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct (2023, methodology): “Our baselines encompass the following models: StarCoder [11], LLaMa [8], CodeGen [13], 5https://github.com/sahil280114/codealpaca CodeGeeX [14], CodeT5+[18], and InCoder[15].”
- A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2023, background): “Step1: Prompt Rewrite Step 2: Scenario Nest Jailbreak Prompt encompassing chatbots, code optimization, data augmentation, data annotation, and tool utilization (Liu et al., 2023a; Zheng et al., 2023; Sahu et al., 2023; He et al., 2023; Liu et al., 2023d).”
- Evaluating Large Language Models in Class-Level Code Generation (2024, background): “To date, researchers have proposed various LLMs [10, 13, 19, 27, 30, 42, 49, 53, 65, 75, 76] (such as GPT-4 [53], WizardCoder [49], and Instruct-CodeGen [10]) by training large models with over billions of parameters on massive general or code-specific corpora and instructions.”
- Qwen Technical Report (2023, methodology): “Our analysis reveals that specialized models, specifically CODE-QWEN and CODE-QWEN-CHAT, significantly outperform previous baselines with similar parameter counts, such as OCTOGEEX (Muennighoff et al., 2023), InstructCodeT5+ (Wang et al., 2023d), and CodeGeeX2 (Zheng et al., 2023).”
- DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence (2024, methodology): “We compare DeepSeek-Coder with the previous state-of-the-art large language models: • CodeGeeX2 (Zheng et al., 2023) represents the second generation of the multilingual code generation model CodeGeeX.”
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, unknown): “…et al., 2022], GPT-4 [OpenAI, 2023], Codex [Chen et al., 2021], LLaMA [Touvron et al., 2023], CodeGen [Nijkamp et al., 2023], Incoder [Fried et al., 2022], GPT-J [Wang and Komatsuzaki, 2021], GPT-Neo and GPT-NeoX [Black et al., 2022], MIM [Nguyen et al., 2023], CodeGeeX [Zheng et al., 2023].”
- TinyLlama: An Open-Source Small Language Model (2024, methodology): “• HumanEval (Zheng et al., 2023): This task is used to measure a model’s programming capabilities.”
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, background): “X [76] (C++, Javascript and Go), CodeContests [33] (C++ and Java) and MultiPL-E [9] (extending H UMAN E VAL and MBPP to 18 programming languages).”
- MetaGPT: Meta Programming for Multi-Agent Collaborative Framework (2023, methodology): “We conduct a comparative evaluation by comparing recent domain-specific LLMs in the code generation field, including AlphaCode [28], Incoder [43], CodeGeeX [44], and CodeX [24] and CodeT [45], with general domain LLMs such as PaLM [46], GPT-4 [47], and MetaGPT.”
- LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023, background): “Recently, there have been a plethora of open-source LLMs demonstrating strong capabilities in writing code, e.g., CodeGen [103], StarCoder [94], and CodeGeeX [104].”
- Competition-Level Problems are Effective LLM Evaluators (2023, background): “Meanwhile, research on open-source code LLMs is also thriving, such as CodeLlama (Rozière et al., 2023), StarCoder (Li et al., 2023), CodeGeeX (Zheng et al., 2023), CodeFuse (Di et al., 2023), WizardCoder (Luo et al., 2023) and Lemur (Xu et al., 2023).”
- Teaching Large Language Models to Self-Debug (2023, background): “In particular, recent large language models have demonstrated a significant leap in improvement over prior deep neural networks (Chen et al., 2021a; Nijkamp et al., 2023; Zheng et al., 2023; Xu et al., 2022).”