PanGu-Coder: Program Synthesis with Function-Level Language Modeling

By Fenia Christopoulou, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhong-yi Li, Qi Zhang, M. Xiao, Bo Shen, Lin Li and 12 more. 2022

87 citations recorded by the source corpus.

We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i.e. the synthesis of programming language solutions given a natural language problem description. We train PanGu-Coder using a two-stage strategy: the first stage employs Causal Language Modelling (CLM) to pre-train on raw programming language data, while the second stage uses a combination of Causal Language Modelling and Masked Language Modelling (MLM) training objectives that focus on the downstream task of text-to-code generation and train on loosely curated pairs of natural language program definitions and code functions. Finally, we discuss PanGu-Coder-FT, which is fine-tuned on a combination of competitive programming problems and code with continuous integration tests. We evaluate PanGu-Coder with a focus on whether it generates functionally correct programs and demonstrate that it achieves equivalent or better performance than similarly sized models, such as CodeX, while attending a smaller context window and training on less data.

Subfields: Artificial Intelligence, Information Systems, Software, Large Language Models, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

In Large Language Models

In Code Generation & Program Synthesis

Citations in the corpus

SantaCoder: don't reach for the stars! (2023, methodology): “…have focused on using decoder-only models with a causal language modeling objective (Chen et al., 2021; Austin et al., 2021; Nijkamp et al., 2022; Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022b; Athiwaratkun et al., 2022), while other studies have investigated encoder (Feng et…”
The Stack: 3 TB of permissively licensed source code (2022, background): “Several groups have explored decoder-only models with a causal language modeling objective (Chen et al., 2021; Austin et al., 2021; Nijkamp et al., 2022; Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022) and generally found that larger models are increasingly capable of synthesizing p”
RLTF: Reinforcement Learning from Unit Test Feedback (2023, background): “…(Feng et al., 2020), decoder-only CodeGPT (Lu et al., 2021), CodeGen (Nijkamp et al., 2022), PaLM-Coder (Chowdhery et al., 2022), PanGu-Coder (Christopoulou et al., 2022), CodeGeex (Zheng et al., 2023), and SantaCoder (Allal et al., 2023), as well as encoder-decoder transformer architectures…”
Evaluating Large Language Models in Class-Level Code Generation (2024, background): “Code LLMs , which are large models mainly trained with massive code-specific corpora and instructions, often have better capability than general LLMs in code generation tasks [24, 49, 70].”
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024, unknown): “Additionally, specialized LLMs have been pre-trained for generating code, e.g., CodeGen (Ni-jkamp et al., 2022), PanGu-Coder (Christopoulou et al., 2022), CodeGeeX (Zheng et al., 2023), CodeFuse (Di et al., 2023), CodeT5+ (Wang et al., 2023d), AlphaCode (Li et al., 2022), In-Coder (Fried et al.,…”