CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation

By Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Shuiguang Deng and 1 more. 2023

66 citations recorded by the source corpus.

Large Language Models (LLMs) have demonstrated remarkable performance on assisting humans in programming and facilitating programming automation. However, existing benchmarks for evaluating the code understanding and generation capacities of LLMs suffer from severe limitations. First, most benchmarks are insufficient as they focus on a narrow range of popular programming languages and specific tasks, whereas real-world software development scenarios show a critical need to implement systems with multilingual and multitask programming environments to satisfy diverse requirements. Second, most benchmarks fail to consider the actual executability and the consistency of execution results of the generated code. To bridge these gaps between existing benchmarks and expectations from practical applications, we introduce CodeScope, an execution-based, multilingual, multitask, multidimensional evaluation benchmark for comprehensively measuring LLM capabilities on coding tasks. CodeScope covers 43 programming languages and eight coding tasks. It evaluates the coding performance of LLMs from three dimensions (perspectives): length, difficulty, and efficiency. To facilitate execution-based evaluations of code generation, we develop MultiCodeEngine, an automated code execution engine that supports 14 programming languages. Finally, we systematically evaluate and analyze eight mainstream LLMs and demonstrate the superior breadth and challenges of CodeScope for evaluating LLMs on code understanding and generation tasks compared to other benchmarks. The CodeScope benchmark and code are publicly available at https://github.com/WeixiangYAN/CodeScope.

Subfields: Artificial Intelligence, Information Systems, Materials Chemistry, Large Language Models, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Armando Solar-Lezama
2. Koushik Sen
3. Naman Jain
4. Tianjun Zhang
5. Wen-Ding Li
6. Ion Stoica
7. Swarat Chaudhuri
8. Rishabh Singh

In Information Systems

1. Armando Solar-Lezama
2. Koushik Sen
3. Naman Jain
4. Tianjun Zhang
5. Ion Stoica
6. Rishabh Singh
7. Mosharaf Chowdhury
8. Charles Sutton

In Large Language Models

1. Armando Solar-Lezama
2. Naman Jain
3. Tianjun Zhang
4. Ion Stoica

In Evaluation & Benchmarks

1. Armando Solar-Lezama
2. Naman Jain
3. Tianjun Zhang
4. Ion Stoica

In Code Generation & Program Synthesis

1. Armando Solar-Lezama
2. Koushik Sen
3. Rishabh Singh

Citations in the corpus

• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “A few benchmarks specifically measure competitive programming, such as APPS (Hendrycks et al., 2021), CodeContests (Li et al., 2022), CodeScope (Yan et al., 2023), xCodeEval (Khan et al., 2023), and LeetCode-Hard (Shinn et al., 2023), and TACO (Li et al., 2023c).”