CodeBERT: A Pre-Trained Model for Programming and Natural Languages

By Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting-Wei Liu, Daxin Jiang and 1 more. 2020

4,078 citations recorded by the source corpus.

We present CodeBERT, a bimodal pre-trained model for programming language (PL) and natural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language code search, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both “bimodal” data of NL-PL pairs and “unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NLPL probing.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Natural Language Processing

Citations in the corpus

Competition-level code generation with AlphaCode (2022, methodology): “, 2020) have created a surge of interest in using transformer models for code retrieval, translation and generation (Chen et al., 2021; Clement et al., 2020; Feng et al., 2020), making significant progress on program synthesis challenges.”
Program Synthesis with Large Language Models (2021, methodology): “…models for code have shown to be effective for transfer learning across software engineering tasks, including CuBERT [Kanade et al., 2020], CodeBERT [Feng et al., 2020], PyMT5 [Clement et al., 2020], code2vec [Alon et al., 2019], and other T5 models trained on code [Mastropaolo et al., 2021].”
Evaluating Large Language Models Trained on Code (2021, neutral): “Popular language modeling objectives like masked language modeling (Devlin et al., 2018) and span prediction (Raffel et al., 2020) have also been adapted to train their programming counterparts CodeBERT (Feng et al., 2020) and PyMT5 (Clement et al., 2020).”
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (2020, background): “…% 47.4 47.4 49.1 The idea of using unlabeled data to networks has been shown effective across many ﬁelds, including computer vision (Vincent et al., 2008; Erhan et al., 2010), NLP (Peters et al., 2018; Devlin et al., 2019), graphs (Hu et al., 2020), and programming languages (Feng et al., 2020).”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “, decoder-only language models [10, 18, 35, 68, 46], encoder-only masked language models [24, 37], and encoder-decoder models [4, 44, 54, 64].”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, methodology): “Prior works, such as CuBERT (Kanade et al., 2020), CodeBERT (Feng et al., 2020), PyMT5 (Clement et al., 2020), and CodeT5 (Wang et al., 2021), have applied transformers towards code understanding but these mostly focus on code retrieval, classification, and program repair.”
Code Llama: Open Foundation Models for Code (2023, neutral): “…2023; Nguyen et al., 2023), many tasks related to code understanding or synthesis have been addressed since the early 2020s with NLP models adapted for code (Raffel et al., 2020; Feng et al., 2020; Guo et al., 2021; Wang et al., 2021; Ahmad et al., 2021), also see the survey by Xu & Zhu (2022).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “r Experience Designer for Google, said, “Google is no longer just ten blue links on a page, those days are long gone” [ReadWrite, 2010]. 3And indeed, programming languages as well [Alon et al., 2020, Feng et al., 2020]! 4https://www.blog.google/products/search/search-language-understanding-bert/ 5ht”
StarCoder: may the source be with you! (2023, methodology): “Many neural architectures developed in NLP were also applied successfully to code, including encoder-only models for producing code representations (Feng et al., 2020; Kanade et al., 2020) and encoder-decoder models for translation, editing, summarization, and language-to-code tasks (Wang et al.,…”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “Two inﬂuential early models were CodeBERT [16] and PyMT [17].”
SantaCoder: don't reach for the stars! (2023, methodology): “…2022; Christopoulou et al., 2022; Izadi et al., 2022; Xu et al., 2022b; Athiwaratkun et al., 2022), while other studies have investigated encoder (Feng et al., 2020a; Kanade et al., 2020) and encoder-decoder architectures (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al.,…”
The Stack: 3 TB of permissively licensed source code (2022, methodology): “Researchers have also investigated encoder masked language models (Feng et al., 2020; Kanade et al., 2020) and encoder-decoder architectures with various training objectives (Li et al., 2022; Ahmad et al., 2021; Wang et al., 2021; Roziere et al., 2021).”
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, background): “Examples include OpenAI’s Codex [3, 5], AI21’s Jurrasic J1 [13,14], Salesforce CodeGen [15] and CodeBERT [16].”
A systematic evaluation of large language models of code (2022, background): “CodeBERT (125M) (Feng et al., 2020) and CuBERT (345M) (Kanade et al., 2020) are examples of such models in code.”
Unified Pre-training for Program Understanding and Generation (2021, methodology): “Later, BERT’s task-independent pre-training approach is rigorously studied (Devlin et al., 2019; Liu et al., 2019; Solaiman et al., 2019; Feng et al., 2020; Sun et al., 2019; Li et al., 2020).”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “To carry out code understanding tasks like clone detection, defect detection, cloze test, and code search, we use CodeBERT [15] as our encoder.”
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, methodology): “…(Karaivanov, Raychev, and Vechev 2014; Oda et al. 2015; Barone and Sennrich 2017; Chen, Liu, and Song 2018; Kanade et al. 2019; Husain et al. 2019; Feng et al. 2020; Dinella et al. 2020; Lachaux et al. 2020), different automatic evaluation methods for code synthesis are leveraged, including…”