CodeT5+: Open Code Large Language Models for Code Understanding and Generation
By Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, Steven C. H. Hoi, S. Hoi. 2023
742 citations recorded by the source corpus.
Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. However, existing code LLMs have two main limitations in terms of architecture and pretraining tasks. First, they often adopt a specific architecture (encoder-only or decoder-only) or rely on a unified encoder-decoder network for different downstream tasks. The former paradigm is limited by inflexibility in applications while in the latter, the model is treated as a single system for all tasks, leading to suboptimal performance on a subset of tasks. Secondly, they often employ a limited set of pretraining objectives which might not be relevant to some downstream tasks and hence result in substantial performance degrade. To address these limitations, we propose ``CodeT5+'', a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks. Such flexibility is enabled by our proposed mixture of pretraining objectives to mitigate the pretrain-finetune discrepancy. These objectives cover span denoising, contrastive learning, text-code matching, and causal LM pretraining tasks, on both unimodal and bimodal multilingual code corpora. Furthermore, we propose to initialize CodeT5+ with frozen off-the-shelf LLMs without training from scratch to efficiently scale up our models, and explore instruction-tuning to align with natural language instructions. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning. We observe state-of-the-art (SoTA) model performance on various code-related tasks, such as code generation and completion, math programming, and text-to-code retrieval tasks. Particularly, our instruction-tuned CodeT5+ 16B achieves new SoTA results on HumanEval code generation task against other open code LLMs.
Subfields: Artificial Intelligence, Large Language Models, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Large Language Models
In Code Generation & Program Synthesis
- Huan Sun
- Chun Xia
- Jiawei Liu
- Lingming Zhang
- Yuyao Wang
- Jian-Guang Lou
- William W. Cohen
- Abhijeet Awasthi
Citations in the corpus
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, background): “5-turbo, codegen-16B-mono, codegen-16B-multi, CodeT5+ and XGen.”
- NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, background): “…more capable language models (LMs) of code, e.g., (Chen et al., 2021; Wang & Komatsuzaki, 2021; Austin et al., 2021; Black et al., 2022; Tunstall et al., 2022; Fried et al., 2022; Nijkamp et al., 2023; Li et al., 2023b; Wang et al., 2023; OpenAI, 2023; Luo et al., 2023; Muennighoff et al., 2023).”
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, methodology): “Our goal is to comprehensively evaluate recent and widely used LLMs, both specialized for code generation [46, 70, 18, 2, 52, 38, 64] and general-purpose tasks [49, 48, 12, 60, 63, 5, 26].”
- AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation (2023, methodology): “For example, GraphCodeBERT [Guo et al. , 2020] incorporates graph-based representations, while CodeT5+ [Wang et al. , 2023b] combines the encoder-decoder paradigm with the structural essence of code.”
- A Comprehensive Overview of Large Language Models (2023, positive): “Similarly, CodeT5+ [124] uses an encoder-decoder architecture with multiple training objectives for different tasks, activating the encoder, decoder, or both according to the tasks.”
- Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “Further advancements are evident in CodeT5+ [286], while AlphaCode [156] and CoTexT [222] showcase the architecture’s adaptability to various SE tasks.”
- Large Language Models on Graphs: A Comprehensive Survey (2023, background): “A potential solution is to adopt codes (that can capture structures) to describe the graphs and utilize code LLMs [22].”
- OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024, neutral): “Additionally, specialized LLMs have been pre-trained for generating code, e.g., CodeGen (Ni-jkamp et al., 2022), PanGu-Coder (Christopoulou et al., 2022), CodeGeeX (Zheng et al., 2023), CodeFuse (Di et al., 2023), CodeT5+ (Wang et al., 2023d), AlphaCode (Li et al., 2022), In-Coder (Fried et al., 202”
- WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “Llemma (Azerbayev et al., 2023) and MathPile (Wang et al., 2023c) continue pretraining LLMs with math corpus to improve domain capacity.”
- CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, methodology): “…of assessing integrity and comprehensiveness for code understanding and generation capabilities of LLMs. Firstly, many code LLMs, such as CodeT5+ (Wang et al., 2023b), WizardCoder (Luo et al., 2023), and Code LLaMA (Rozière et al., 2023), employ their own specific single-task evaluation…”
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct (2023, methodology): “CodeGeeX [14], CodeT5+[18], and InCoder[15].”
- Structured Chain-of-Thought Prompting for Code Generation (2023, neutral): “, CodeParrot [29], CodeGen [17], CodeGeeX [40], InCoder [8], StarCoder [15] and CodeT5+ [33].”
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “More recently, (Chen et al., 2023; Wang et al., 2023e) have proposed to tackle more challenging college-level science and math problems.”
- Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “CodeT5+ [252] adopts the CodeT5 [106] model for both the retriever and generator, leveraging only the encoder part in the retrieval process.”
- TheoremQA: A Theorem-driven Question Answering dataset (2023, methodology): “…et al., 2020), Claude (Bai et al., 2022), LLaMA (Touvron et al., 2023), Pythia (Bi-derman et al., 2023), CodeGen (Nijkamp et al., 2022), GLM (Zeng et al., 2022), StarCoder (Li et al., 2023), and CodeT5+ (Wang et al., 2023) Figure 1: The overview of TheoremQA and the prompting strategies adopted.”
- CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (2024, unknown): “CodeT5+ (Wang et al., 2023b) uses a contrastive similarity score to sample negatives from a queue maintained by a momentum encoder (He et al., 2020).”