CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code

By Shuyan Zhou, Uri Alon, Sumit Agarwal, Graham Neubig. 2023

194 citations recorded by the source corpus.

Since the rise of neural natural-language-to-code models (NL->Code) that can generate long expressions and statements rather than a single next-token, one of the major problems has been reliably evaluating their generated output. In this paper, we propose CodeBERTScore: an evaluation metric for code generation, which builds on BERTScore (Zhang et al., 2020). Instead of encoding only the generated tokens as in BERTScore, CodeBERTScore also encodes the natural language input preceding the generated code, thus modeling the consistency between the generated code and its given natural language context as well. We perform an extensive evaluation of CodeBERTScore across four programming languages. We find that CodeBERTScore achieves a higher correlation with human preference and with functional correctness than all existing metrics. That is, generated code that receives a higher score by CodeBERTScore is more likely to be preferred by humans, as well as to function correctly when executed. We release five language-specific pretrained models to use with our publicly available code. Our language-specific models have been downloaded more than 1,000,000 times from the Huggingface Hub. Our code and data are available at https://github.com/neulab/code-bert-score

Subfields: Artificial Intelligence, Information Systems, Materials Chemistry, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Materials Chemistry

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

Citations in the corpus

DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “Therefore, most recent papers that evaluate on real-world programs have to rely on unreliable surface-form metrics (Ren et al., 2020; Chen et al., 2021b; Xu et al., 2022), or similarity scores (Zhou et al., 2023) calculated by code representations (Wang et al., 2022).”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, methodology): “Thus for future work, we would like to add a surface-form-based evaluation for code, such as (Zhou et al., 2023).”
Learning Performance-Improving Code Edits (2023, methodology): “Inspired by recent prompting strategies (Zhang et al., 2023) and offline-rl (Chen et al., 2021a), we introduce performance tags during training by associating each “fast” program with a tag indicating the optimal achievable performance across all solutions in the dataset.”