CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
By Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang and 13 more. 2021
1,545 citations recorded by the source corpus.
Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.
Subfields: Information Systems, Software, Natural Language Processing, Evaluation & Benchmarks, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Chun Xia
- Lingming Zhang
- Uri Alon
- Graham Neubig
- Prem Devanbu
- Charles Sutton
- Miltiadis Allamanis
- Vincent J. Hellendoorn
In Software
- Chun Xia
- Lingming Zhang
- Prem Devanbu
- Toufique Ahmed
- Vincent J. Hellendoorn
- Baishakhi Ray
- Charles Sutton
- Baptiste Rozière
In Natural Language Processing
- Graham Neubig
- Luke S. Zettlemoyer
- Mohit Bansal
- Jianfeng Gao
- Caiming Xiong
- Percy Liang
- Richard Socher
- Kyunghyun Cho
In Evaluation & Benchmarks
- Luke S. Zettlemoyer
- Dan Roth
- Caiming Xiong
- Graham Neubig
- Quoc V. Le
- Thomas Scialom
- Michael Lewis
- Wen-tau Yih
In Signal Processing
- Gabriel Synnaeve
- Yossi Adi
- Quoc V. Le
- Graham Neubig
- Tal Remez
- Jade Copet
- Charles Sutton
- Lingming Zhang
Citations in the corpus
- Program Synthesis with Large Language Models (2021, methodology): “Multi-task benchmarks for these tasks have been collected into CodeXGlue [Lu et al., 2021].”
- Evaluating Large Language Models Trained on Code (2021, background): “Recently, CodeXGLUE (Lu et al., 2021) aggregated several programming benchmarks, making use of the recently proposed CodeBLEU metric (Ren et al., 2020).”
- InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “We also include the performance of the supervised baseline as reported in the CodeXGLUE paper.”
- Code Llama: Open Foundation Models for Code (2023, methodology): “The Python subsection of the CodeXGLUE code summarization benchmark Lu et al. (2021) can be used as an infilling benchmark (Fried et al., 2023; Li et al., 2023) in which a docstring surrounded by triple quotes has to be inserted between the function header and body in a Python function definition.”
- MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, background): “Lu et al. [19] propose a suite of evaluation datasets for ten tasks, including code translation, docstring generation, and code summarization.”
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, methodology): “Specifically, we use the unbiased method to estimate pass@ 𝑘 [7]: where 𝑛 is the total number of generations ( 𝑛 =200 in this work), 𝑘 is the sampling budget (typically 𝑘 ∈ { 1, 10, 100 } ), and 𝑐 is the number of samples that pass all test cases.”
- SantaCoder: don't reach for the stars! (2023, neutral): “CodeXGLUE (Lu et al., 2021) is a set of 14 datasets for evaluating code generation models.”
- The Stack: 3 TB of permissively licensed source code (2022, background): “CodeXGLUE (Lu et al., 2021) is a set of 14 datasets for evaluating code generation models.”
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “We fine-tune CodeT5 on most tasks in the CodeXGLUE benchmark (Lu et al., 2021), including two understanding tasks: code defect detection and clone detection, and generation tasks such as code summarization, generation, translation, and refinement.”
- Unified Pre-training for Program Understanding and Generation (2021, methodology): “GPT-2, CodeGPT-2, and CodeGPT-adapted are GPT-style models.”
- DOBF: A Deobfuscation Pre-Training Objective for Programming Languages (2021, methodology): “In order to evaluate DOBF as a pre-training model, we fine-tune DOBF on TransCoder and on three tasks from CodeXGLUE Lu et al. [2021], a benchmark for programming languages. The data, code and models from CodeXGLUE and TransCoder are available respectively under the MIT and the Creative Commons lice”
- Multi-lingual Evaluation of Code Generation Models (2022, methodology): “To evaluate the code summarization performance, we use smoothed BLEU score as the metrics following the setting in CodeXGLUE (Lu et al., 2021) that compare the generated outputs with the groundtruth docstrings.”
- Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming (2023, neutral): “code operations such as code explanation and search [56, 77].”
- VerilogEval: Evaluating Large Language Models for Verilog Code Generation (2023, background): “However, recent works [21], [26] have argued that such metrics does not correlate well with functional correctness.”
- LongCoder: A Long-Range Pre-trained Language Model for Code Completion (2023, methodology): “To ensure fair comparison with other models, we pre-train LongCoder on the CodeSearchNet dataset using the same next token prediction objective and pre-training setting as baselines (Lu et al., 2021; Guo et al., 2022).”
- PLUR: A Unifying, Graph-Based View of Program Learning, Understanding, and Repair (2021, background): “CodeXGLUE [Lu et al., 2021] further expanded the benchmark to include tasks from code summarization, repair, code generation, and more.”
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X (2023, methodology): “…correctness of the generated code, measuring the performance with pass@k (Kulal et al., 2019), making it real-world useful and also completely different from the string similarity metrics like BLEU (Papineni et al., 2002), and CodeBLEU (Ren et al., 2020; Lu et al., 2021; Zhu et al., 2022).”