Unified Pre-training for Program Understanding and Generation
By Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang. 2021
938 citations recorded by the source corpus.
Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART’s effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., “if“ block inside an “else“ block is equivalent to “else if“ block) that are crucial to program semantics and thus excels even with limited annotations.
Subfields: Artificial Intelligence, Software, Natural Language Processing, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Percy Liang
- Caiming Xiong
- Yejin Choi
- Hannaneh Hajishirzi
In Software
- Chun Xia
- Lingming Zhang
- Prem Devanbu
- Yinlin Deng
- Toufique Ahmed
- Haoran Peng
- Chenyuan Yang
- Gabriel Synnaeve
In Natural Language Processing
- Graham Neubig
- Luke S. Zettlemoyer
- Mohit Bansal
- Mirella Lapata
- Jianfeng Gao
- Percy Liang
- Wen-tau Yih
- Dan Roth
In Code Generation & Program Synthesis
- Prem Devanbu
- Frank F. Xu
- Graham Neubig
- Chun Xia
- Lingming Zhang
- Toufique Ahmed
- Vincent J. Hellendoorn
- Yangruibo Ding
Citations in the corpus
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “, [18, 45, 4]), we evaluate our model zero-shot, with no explicit documentation string supervision.”
- Code Llama: Open Foundation Models for Code (2023, neutral): “…2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et al., 2020; Feng et al., 2020; Guo et al., 2021; Wang et al., 2021; Ahmad et al., 2021), also see the survey by Xu & Zhu (2022).”
- StarCoder: may the source be with you! (2023, methodology): “…were also applied successfully to code, including encoder-only models for producing code representations (Feng et al., 2020; Kanade et al., 2020) and encoder-decoder models for translation, editing, summarization, and language-to-code tasks (Wang et al., 2021; Ahmad et al., 2021; Li et al., 2022).”
- SantaCoder: don't reach for the stars! (2023, background): “…Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022b; Athiwaratkun et al., 2022), while other studies have investigated encoder (Feng et al., 2020a; Kanade et al., 2020) and encoder-decoder architectures (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al., 2021).”
- The Stack: 3 TB of permissively licensed source code (2022, background): “, 2020) and encoder-decoder architectures with various training objectives (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al., 2021).”
- A systematic evaluation of large language models of code (2022, methodology): “, 2021), and PLBART (406M) (Ahmad et al., 2021) use the two objectives mentioned above respectively, and performs well in conditional generation downstream tasks such as code commenting, or natural language to code generation.”
- A Survey on Pretrained Language Models for Neural Code Intelligence (2022, background): “, 2021b) and PLBART (Ahmad et al., 2021), have been leveraged in recent”
- Large Language Models for Software Engineering: A Systematic Literature Review (2023, methodology): “LLMs, such as BERT [263, 333], CodeBERT [141], CodeT5 [219], Codex [64, 115, 297, 297], PLBART [219, 297], T5 [186, 318] and GPT [27, 31, 138, 253, 264, 300, 301], have demonstrated remarkable capabilities in understanding programming languages and generating syntactically correct and contextually r”
- Measuring The Impact Of Programming Language Distribution (2023, background): “, 2019) and in the program synthesis space (Ahmad et al., 2021) have used sampling strategies relying on temperaturescaling.”
- Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, methodology): “Unsupervised pre-training methods for code based on BERT (Kanade et al., 2020; Feng et al., 2020), BART (Ahmad et al., 2021) or other objectives tailored to source code (Guo et al., 2020; Roziere et al., 2021) have shown strong results on benchmarks such as CodeXGLUE (Lu et al., 2021).”
- ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021, methodology): “PLBART (Ahmad et al., 2021) is similar to multi-lingual BART with language tags to perform de-noising pre-training on programming languages.”
- Few-shot training LLMs for project-specific code-summarization (2022, background): “The majority of the foundation models [1, 2, 7, 9, 23] are evaluated on this dataset for the code summarization task.”
- UniXcoder: Unified Cross-Modal Pre-training for Code Representation (2022, methodology): “For encoder-decoder models, we mainly compare the current unified models PLBART (Ahmad et al., 2021) and CodeT5 (Wang et al., 2021).”
- Less training, more repairing please: revisiting automated program repair via zero-shot learning (2022, methodology): “CodeTrans [22], CodeT5 [72] and PLBART [4] are unified encoder-decoder models [63] which uses denoising sequence-to-sequence training objectives to pre-train both the encoder and decoder for various coding tasks.”
- A Survey on Large Language Models for Code Generation (2024, neutral): “To better illustrate this, Copilot generates insecure code in about 40% of the cases, whereas ChatGPT showed that of the 21 programs, only 5 were initially secure.”
- Code Translation with Compiler Representations (2022, background): “Alternative objectives, have been proposed for programming languages (Guo et al., 2020; Lachaux et al., 2021; Ahmad et al., 2021; Wang et al., 2021).”