Execution-based Evaluation for Data Science Code Generation Models

By Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin B. Clement, Nan Duan, Jianfeng Gao. 2022

43 citations recorded by the source corpus.

Code generation models can benefit data scientists’ productivity by automatically generating code from context and text descriptions. An important measure of the modeling progress is whether a model can generate code that can correctly execute to solve the task. However, due to the lack of an evaluation dataset that directly supports execution-based model evaluation, existing work relies on code surface form similarity metrics (e.g., BLEU, CodeBLEU) for model selection, which can be inaccurate. To remedy this, we introduce ExeDS, an evaluation dataset for execution evaluation for data science code generation tasks. ExeDS contains a set of 534 problems from Jupyter Notebooks, each consisting of code context, task description, reference program, and the desired execution output. With ExeDS, we evaluate the execution performance of five state-of-the-art code generation models that have achieved high surface-form evaluation scores. Our experiments show that models with high surface-form scores do not necessarily perform well on execution metrics, and execution-based metrics can better capture model code generation errors. All the code and data will be released upon acceptance.

Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Information Systems and Management

In Code Generation & Program Synthesis

Citations in the corpus

Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “The predominant approach is to create hand-written test cases (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021; Lai et al., 2022; Huang et al., 2022) for the canonical code solutions.”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, background): “, 2022), or a limited number of contextually dependent problems (Huang et al., 2022), rather than having multiple, related tasks such as in Fig.”
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023, background): “…Metrics To alleviate previous issues, execution-based evaluation counts a generated code snippet as correct if it produces the required outputs when run with given inputs (Chen et al., 2021; Athiwaratkun et al., 2022; Li et al., 2022; Wang et al., 2022; Lai et al., 2022; Huang et al., 2022).”
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, methodology): “This execution-based paradigm has been widely applied to evaluate benchmarks in Data Science domain like DSP (Chandel et al., 2022), DS-1000 (Lai et al., 2022) and Exe-DS (Huang et al., 2022) as well as general code generation benchmarks in single-language settings such as HumanEval (Chen et al., 20”
A Survey on Large Language Models for Code Generation (2024, unknown): “For example, as one token defines one unit of text (e.g. a word or a small piece of a sentence), Llama 3.1 models were trained on over 15 trillion tokens, in which Llama 3.1 - 8B required 7 million GPU hours while Llama 3.1 - 405B required approximately 31 million GPU hours.”
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “In addition, most of the related works (Hendrycks et al., 2021; Lai et al., 2023; Huang et al., 2022; Nijkamp et al., 2023; Chandel et al., 2022) focused on code generation tasks with target as Python when evaluating model performance, mainly because setting up a Python execution environment is…”