Execution-Based Evaluation for Open-Domain Code Generation

By Zhiruo Wang, Shuyan Zhou, Daniel Fried, Graham Neubig. 2022

119 citations recorded by the source corpus.

To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution. Our NL-Code pairs are harvested from StackOverflow forums to encourage natural and practical coding queries. Moreover, ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian. ODEX unveils intriguing behavioral differences among top-performing code language models (LM). While CODEX achieves better overall results, CODEGEN improves effectively via scaling -- CODEGEN 6.1B performs comparably with CODEX 12B. Both models show substantial gaps between open and closed domains, but CODEGEN gaps tend to decrease with model size while CODEX gaps increase. We release ODEX to facilitate research into open-domain problems for the code generation community.

Subfields: Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Code Generation & Program Synthesis

Citations in the corpus

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “…APIs or general software engineering tasks, such as JuICe (Agashe et al., 2019), APIBench (Patil et al., 2023), RepoBench (Liu et al., 2023c), ODEX (Wang et al., 2022b), SWE-Bench (Jimenez et al., 2023), GoogleCodeRepo (Shrivastava et al., 2023), RepoEval (Zhang et al., 2023a), and Cocomic-Data…”
LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, background): “Problem decomposition has been identified as a key approach for improving the reasoning capabilities of models (Zhou et al., 2022; Wang et al., 2023).”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, methodology): “To alleviate this limitation, ODEX (Wang et al., 2022b) provides execution-based evaluation via human-written test cases of diversiﬁed Python libraries.”
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “Recently, open-domain coding benchmarks [58, 32] have been built to challenge code LLMs on application-specific scenarios by using specific sets of tools and libraries.”
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023, background): “…Metrics To alleviate previous issues, execution-based evaluation counts a generated code snippet as correct if it produces the required outputs when run with given inputs (Chen et al., 2021; Athiwaratkun et al., 2022; Li et al., 2022; Wang et al., 2022; Lai et al., 2022; Huang et al., 2022).”
WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, background): “This evaluation is not only more reliable [40, 5, 33] than comparing the vanilla action sequences [25, 7] but also accommodates a range of potential valid paths to achieve the same goal, which is a ubiquitous phenomenon in sufficiently complex tasks.”
A Survey on Large Language Models for Code Generation (2024, unknown): “…Language Understanding) model, which lever-Figure 3: How transformer models work [10] Figure 4: Notable released LLMs timeline ages LLMs to analyze and extract valuable insights from customer feedback, social media posts, etc., allowing them to make decisions based on this information [14].”
A Survey of Large Language Models (2023, unknown): “Code Synthesis APPS [287], HumanEval [89], MBPP [140], CodeContest [97], MTPB [76], DS-1000 [288], ODEX [289]”