GraphCodeBERT: Pre-training Code Representations with Data Flow
By Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Shujie Liu, Long Zhou, Nan Duan, Jian Yin and 9 more. 2020
1,778 citations recorded by the source corpus.
Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc. However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process. We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables. Such a semantic-level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks. One is to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model in an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement. Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.
Subfields: Artificial Intelligence, Information Systems, Software, Natural Language Processing, Graph Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Graham Neubig
- Luke S. Zettlemoyer
- Percy Liang
- Furu Wei
- Caiming Xiong
In Information Systems
In Software
In Natural Language Processing
In Graph Learning
- Petros Maniatis
- Wenhan Xiong
- Prem Devanbu
- Ge Li
- Daniel Tarlow
- Steven C. H. Hoi
- Nicolas Usunier
- Vincent J. Hellendoorn
Citations in the corpus
- Code Llama: Open Foundation Models for Code (2023, methodology): “The Python subsection of the CodeXGLUE code summarization benchmark Lu et al. (2021) can be used as an infilling benchmark (Fried et al., 2023; Li et al., 2023) in which a docstring surrounded by triple quotes has to be inserted between the function header and body in a Python function definition.”
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “For the clone detection task, our CodeT5 models achieve comparable re-sults to the SOTA GraphCodeBERT and PLBART models.”
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “Then we build the data-flow graph [25] for each function, which represents the dependency between two variables and provides valuable semantic information for code understanding.”
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis (2020, methodology): “We use data-flow (Guo et al. 2020) to represent a source code as a graph, in which nodes represent variables and edges represent where the value of each variable comes from.”
- DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, background): “As in natural language, pre-training was shown to be effective for source code [20, 24, 41].”
- AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Following BioBERT in biomedical domain, models like AleaseBERT [32], RoBERTaNews [34], GraphCodeBERT [38], CoText [39], CodeGPTadapted [35], NetBERT [41], MathBERT [44], TOD-BERT”
- Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “Previous research on code representations has shown the benefits of incorporating data flow, which captures the semantic-level structure of code and represents the relationship between variables in terms of “whether-value-comes-from” [152].”
- PLUR: A Unifying, Graph-Based View of Program Learning, Understanding, and Repair (2021, methodology): “It also provided several baseline model implementations of different characteristics (e.g., CodeBERT [Feng et al., 2020] similar to BERT [Devlin et al., 2019], GraphCodeBERT [Guo et al., 2021] incorporating data-flow edges, etc.), each adapted to a particular subset of task types (e.g., a GPT-based m”
- ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021, background): “CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo et al., 2020) are two pre-training models for programming languages.”
- An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2023, neutral): “When setting temperature=0, the code candidates generated from the Concise prompt are more deterministic than our Base prompt, while those code candidates generated from the CoT prompt suffer from higher randomness than our Base prompt.”
- Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, methodology): “…demonstrates that, by scaling the amount of data and computational resources, the language model can acquire high-quality code representations directly from sequences of code tokens, without the need for incorporating human knowledge about the structural information of code (Guo et al., 2021).”
- DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection (2023, methodology): “GraphCodeBERT [10] also uses the CodeSearchNet [11] training datasets.”
- TU_DBS in the ARQMath Lab 2021, CLEF (2021, methodology): “One possible improvement could be the prediction of relationships between nodes in these trees as it was done in a similar way for Programming Language Understanding using data flow graphs [26].”
- CoSQA: 20,000+ Web Queries for Code Search and Question Answering (2021, background): “There are also ways to exploit code structures to learn better representations for code search (Wan et al., 2019; Haldar et al., 2020; Guo et al., 2020).”
- Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, methodology): “, 2021) or other objectives tailored to source code (Guo et al., 2020; Roziere et al., 2021) have shown strong results on benchmarks such as CodeXGLUE (Lu et al.”