CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

By Yue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. Hoi. 2021

2,385 citations recorded by the source corpus.

Pre-trained models for Natural Languages (NL) like BERT and GPT have been recently shown to transfer well to Programming Languages (PL) and largely benefit a broad set of code-related tasks. Despite their success, most current methods either rely on an encoder-only (or decoder-only) pre-training that is suboptimal for generation (resp. understanding) tasks or process the code snippet in the same way as NL, neglecting the special characteristics of PL such as token types. We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code. Our code and pre-trained models are released at https://github.com/salesforce/CodeT5.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Graham Neubig
4. Luke S. Zettlemoyer
5. Percy Liang
6. Caiming Xiong
7. Furu Wei
8. Yoshua Bengio

In Information Systems

1. Chun Xia
2. Lingming Zhang
3. Uri Alon
4. Michael Pradel
5. Yinlin Deng
6. Yanqiao Zhu
7. Yichen Xu
8. Graham Neubig

In Natural Language Processing

1. Graham Neubig
2. Mirella Lapata
3. Luke S. Zettlemoyer
4. Percy Liang
5. Mohit Bansal
6. Jianfeng Gao
7. Richard Socher
8. Caiming Xiong

In Signal Processing

1. Yossi Adi
2. Gabriel Synnaeve
3. Jade Copet
4. Tal Remez
5. Michael Pradel
6. Arjun Guha
7. Itai Gat
8. Lingming Zhang

Citations in the corpus

• InCoder: A Generative Model for Code Infilling and Synthesis (2022, neutral): “, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
• Code Llama: Open Foundation Models for Code (2023, background): “In parallel, models specifically trained or fine-tuned for code understanding and program synthesis from natural language prompts emerged with LLMs such as Codex (Chen et al., 2021), CodeT5 (Wang et al., 2021), InCoder (Fried et al., 2023), AlphaCode (Li et al., 2022), CodeGen (Nijkamp et al., 2023b”
• SantaCoder: don't reach for the stars! (2023, background): “…Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022b; Athiwaratkun et al., 2022), while other studies have investigated encoder (Feng et al., 2020a; Kanade et al., 2020) and encoder-decoder architectures (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al., 2021).”
• The Stack: 3 TB of permissively licensed source code (2022, methodology): “, 2020) and CodeT5 (Wang et al., 2021) used this dataset for pre-training.”
• A systematic evaluation of large language models of code (2022, methodology): “In code, CodeT5 (220M) (Wang et al., 2021), and PLBART (406M) (Ahmad et al., 2021) use the two objectives mentioned above respectively, and performs well in conditional generation downstream tasks such as code commenting, or natural language to code generation.”
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “These include CodeT5 (Wang et al., 2021, 2023), CodeGen (Nijkamp et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023b), AlphaCode (Li et al., 2022), InCoder (Fried et al., 2022), and CodeGeeX (Zheng et al., 2023).”
• LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, background): “Recently, pre-trained code language models (Chen et al., 2021a; Wang et al., 2021; Fried et al., 2022; Nijkamp et al., 2022; OpenAI, 2022) have demonstrated surprisingly strong performance in this problem across programming languages (Lin et al., 2018; Yu et al., 2018; Austin et al., 2021; Cobbe et…”
• A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “CodeT5 [193] proposes a pre-trained encoder-decoder Transformer model which better captures contextualized semantic information from code.”
• BloombergGPT: A Large Language Model for Finance (2023, neutral): “…et al., 2022; Tafjord and Clark, 2021; Smith et al., 2022; Scao et al., 2022; Taylor et al., 2022; Lin et al., 2022; Soltan et al., 2022; Thoppilan et al., 2022; Bao et al., 2022; Sanh et al., 2022; Roller et al., 2021; Glaese et al., 2022; Wang et al., 2021; Peng et al., 2022, among many others).”
• AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation (2023, background): “One of the most pivotal applications for these LLMs is code generation for downstream tasks, where they play a vital role in aiding developers in creating software [Feng et al. , 2020; Wang et al. , 2021; Wang et al. , 2023b; Nijkamp et al. , 2023b; Nijkamp et al. , 2023a; Li et al. , 2023b].”
• DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection (2023, methodology): “We select four model achitectures from the T5 family: T5 Base [23], CodeT5 Base, CodeT5 Small [27], and NatGen [3].”
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation (2022, methodology): “CodeT5 (Wang et al., 2021) adapts the T5 (Raffel et al.”
• A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, background): “, 2023), several works have proposed programming-focused language models (Chen et al., 2021a; Feng et al., 2020; Li et al., 2022; Wang et al., 2021) and their benchmarks (Austin et al.”
• DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence (2024, background): “However, a major challenge in this field is the performance gap between open-source models (Li et al., 2023; Nijkamp et al., 2022; Roziere et al., 2023; Wang et al., 2021) and closed-source models (Gemini Team, 2023; OpenAI, 2023).”