CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

By Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, M. Zhou, Ambrosio Blanco, Shuai Ma. 2020

902 citations recorded by the source corpus.

Evaluation metrics play a vital role in the growth of an area as it defines the standard of distinguishing between good and bad models. In the area of code synthesis, the commonly used evaluation metric is BLEU or perfect accuracy, but they are not suitable enough to evaluate codes, because BLEU is originally designed to evaluate the natural language, neglecting important syntactic and semantic features of codes, and perfect accuracy is too strict thus it underestimates different outputs with the same semantic logic. To remedy this, we introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the strength of BLEU in the n-gram match and further injects code syntax via abstract syntax trees (AST) and code semantics via data-flow. We conduct experiments by evaluating the correlation coefficient between CodeBLEU and quality scores assigned by the programmers on three code synthesis tasks, i.e., text-to-code, code translation, and code refinement. Experimental results show that our proposed CodeBLEU can achieve a better correlation with programmer assigned scores compared with BLEU and accuracy.

Subfields: Information Systems, Software, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Evaluation & Benchmarks

Citations in the corpus

Competition-level code generation with AlphaCode (2022, background): “Early work evaluated performance by measuring how well the generated code matched the ground truth reference code at the token level, syntax tree level, or full program level (Ren et al., 2020).”
Evaluating Large Language Models Trained on Code (2021, background): “For instance, Ren et al. (2020) ﬁnds that BLEU has problems capturing semantic features speciﬁc to code, and suggests several semantic modiﬁcations to the score.”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, methodology): “We build the free CodeGeeX extension in several IDEs, currently including Visual Studio Code, JetBrains, and Tencent Cloud Studio (a Web IDE).”
Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “CodeBLEU CodeBLEU (Ren et al., 2020) is speciﬁcally designed for code evaluation, by jointly considering the surface-form match, syntax similarly, and semantic data ﬂows.”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “We evaluate it with BLEU-4, exact match (EM) accuracy, and CodeBLEU (Ren et al., 2020) that considers syntactic and semantic matches based on the code structure in addition to the n-gram match.”
Unified Pre-training for Program Understanding and Generation (2021, background): “CodeBLEU is a metric for measuring the quality of the synthesized code (Ren et al., 2020).”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation (2021, methodology): “The CodeBLEU score is used as the overall score.”
GitHub Copilot AI pair programmer: Asset or Liability? (2022, background): “[43] introduces a new metric, called CodeBLEU, that measures the BLEU score on syntax and semantics of codes.”
Execution-based Evaluation for Data Science Code Generation Models (2022, background): “Code generation benchmarks are predominantly evaluated by matching code surface forms(Papineni et al., 2002; Lin, 2004; Ren et al., 2020).”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X (2023, methodology): “, 2022) only consist of programming problems in a single language and 2) existing multilingual datasets (Ren et al., 2020; Lu et al., 2021; Zhu et al., 2022) use string similarity metrics like BLEU (Papineni et al.”
A Survey on Pretrained Language Models for Neural Code Intelligence (2022, methodology): “For generative tasks, we measure the performance in terms of the textual and structural similarity with ROUGE, BLEU, and CodeBLEU (Papineni et al., 2002; Lin, 2004; Ren et al., 2020), or test the functional correctness of the generated programs (Chen et al., 2021).”
Programming Puzzles (2021, methodology): “To evaluate generation, they use CodeBLEU [47] which relies on ASTs and other code-specific aspects.”
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation (2022, background): “[27] proposes a code generation-specific formulation of this metric.”