Pre-trained Contextual Embedding of Source Code

By Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, Kensen Shi. 2019

83 citations recorded by the source corpus.

The source code of a program not only serves as a formal description of an executable task, but it also serves to communicate developer intent in a human-readable form. To facilitate this, developers use meaningful identifier names and natural-language documentation. This makes it possible to successfully apply sequence-modeling approaches, shown to be effective in natural-language processing, to source code. A major advancement in natural-language understanding has been the use of pre-trained token embeddings; BERT and other works have further shown that pre-trained contextual embeddings can be extremely powerful and can be fine-tuned effectively for a variety of downstream supervised tasks. Inspired by these developments, we present the first attempt to replicate this success on source code. We curate a massive corpus of Python programs from GitHub to pre-train a BERT model, which we call Code Understanding BERT (CuBERT). We also pre-train Word2Vec embeddings on the same dataset. We create a benchmark of five classification tasks and compare fine-tuned CuBERT against sequence models trained with and without the Word2Vec embeddings. Our results show that CuBERT outperforms the baseline methods by a margin of 2.9-22%. We also show its superiority when fine-tuned with smaller datasets, and over fewer epochs. We further evaluate CuBERT's effectiveness on a joint classification, localization and repair task involving prediction of two pointers.

Subfields: Information Systems, Software, Representation & Self-Supervised Learning, Code Generation & Program Synthesis, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Nan Duan
2. Daya Guo
3. Neel Sundaresan
4. Alexey Svyatkovskiy
5. Duyu Tang
6. Jian Yin
7. Colin B. Clement
8. Ting-Wei Liu

In Software

1. Neel Sundaresan
2. Alexey Svyatkovskiy
3. Daya Guo
4. Nan Duan
5. Jian Yin
6. Colin B. Clement
7. Shao Kun Deng
8. Dawn Drain

In Representation & Self-Supervised Learning

1. Ting-Wei Liu
2. Bing Qin
3. Duyu Tang
4. Joseph E. Gonzalez
5. Tianjun Zhang
6. M. Zhou
7. Neel Sundaresan
8. Ajay Jain

In Code Generation & Program Synthesis

1. Jian Yin
2. Neel Sundaresan
3. Shao Kun Deng
4. Colin B. Clement
5. Shengyu Fu
6. Yanlin Wang
7. Daya Guo
8. Michele Tufano

In Signal Processing

1. Long Zhou
2. M. Zhou
3. Jian Yin
4. Neel Sundaresan
5. Bing Qin
6. Ajay Jain
7. Qun Liu
8. Pieter Abbeel

Citations in the corpus

• CodeBERT: A Pre-Trained Model for Programming and Natural Languages (2020, methodology): “A concurrent work (Kanade et al., 2019) uses masked language modeling and next sentence prediction as the objective to train a BERT model on Python source codes, where a sentence is a logical code line as defined by the Python standard.”
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, methodology): “…translation and code change prediction (Karaivanov, Raychev, and Vechev 2014; Oda et al. 2015; Barone and Sennrich 2017; Chen, Liu, and Song 2018; Kanade et al. 2019; Husain et al. 2019; Feng et al. 2020; Dinella et al. 2020; Lachaux et al. 2020), different automatic evaluation methods for code…”
• GraphCodeBERT: Pre-training Code Representations with Data Flow (2020, background): “Existing works (Kanade et al., 2019; Karampatsis & Sutton, 2020; Feng et al., 2020; Svyatkovskiy et al., 2020; Buratti et al., 2020) regard a source code as a sequence of tokens and pre-train models on source code to support code-related tasks such as code search, code completion, code summarization”
• Contrastive Code Representation Learning (2020, methodology): “Unlike the original BERT paper which cuBERT (Kanade et al., 2020) is based on, hyperparameters from RoBERTa have been found to produce better results during pre-training.”
• PanGu-Coder: Program Synthesis with Function-Level Language Modeling (2022, methodology): “Vision [65, 18, 15, 13, 64, 25], Proteins [11], Time Series [71, 72, 55] and Code [42, 70, 26, 31, 54].”
• Generating bug-fixes using pretrained transformers (2021, methodology): “Kanade et al. (Kanade et al., 2019) pretrain a BERT-style model “CuBERT” on a larger Python dataset and then finetune on a related suite of synthetic bugs, achieving over 90% accuracy.”
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation (2022, methodology): “In contrast, other works (Kanade et al., 2019; Feng et al., 2020) pre-train a bidirectional Transformer encoder on source code, which significantly improves the”
• CoRet: Improved Retriever for Code Editing (2025, unknown): “Different techniques have been applied to learning representation such as masked language modelling (MLM) (Feng et al., 2020; Li et al., 2023), next token prediction (Kanade et al., 2020; Li et al., 2023) and contrastive learning (Guo et al., 2022; Zhang et al., 2022, 2024a).”