DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
By Yuhang Lai, Chengxi Li, Tianyi Zhang, Ruiqi Zhong, Luke S. Zettlemoyer, Scott Yih, Daniel Fried, Si-yi Wang, Tao Yu. 2022
563 citations recorded by the source corpus.
We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. Compared to prior works, DS-1000 incorporates three core features. First, our problems reflect diverse, realistic, and practical use cases since we collected them from StackOverflow. Second, our automatic evaluation is highly specific (reliable) -- across all Codex-002-predicted solutions that our evaluation accept, only 1.8% of them are incorrect; we achieve this with multi-criteria metrics, checking both functional correctness by running test cases and surface-form constraints by restricting API usages or keywords. Finally, we proactively defend against memorization by slightly modifying our problems to be different from the original StackOverflow source; consequently, models cannot answer them correctly by memorizing the solutions from pre-training. The current best public system (Codex-002) achieves 43.3% accuracy, leaving ample room for improvement. We release our benchmark at https://ds1000-code-gen.github.io.
Subfields: Artificial Intelligence, Information Systems, Materials Chemistry, Evaluation & Benchmarks, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Caiming Xiong
- Yoshua Bengio
- Noah A. Smith
In Information Systems
In Materials Chemistry
- Siva Reddy
- Graham Neubig
- Caiming Xiong
- Ziyang Luo
- Oleksandr Polozov
- Shafiq R. Joty
- Charles Sutton
- Uri Alon
In Evaluation & Benchmarks
- Graham Neubig
- Dragomir R. Radev
- Shafiq R. Joty
- Chun Xia
- Lingming Zhang
- Caiming Xiong
- Marc Marone
- Shuyan Zhou
In Code Generation & Program Synthesis
Citations in the corpus
- StarCoder: may the source be with you! (2023, methodology): “We confirm the finding by Lai et al. (2022): model performance on HumanEval and MBPP benchmarks does not always correlate with performance on the more realistic DS-1000 benchmarks .”
- SantaCoder: don't reach for the stars! (2023, background): “Many single-language benchmarks for evaluating code completion exist (Kulal et al., 2019; Iyer et al., 2018; Zhong et al., 2017; Yu et al., 2018; Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Austin et al., 2021; Athiwaratkun et al., 2022; Lai et al., 2022).”
- Execution-Based Evaluation for Open-Domain Code Generation (2022, methodology): “The predominant approach is to create hand-written test cases (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021; Lai et al., 2022; Huang et al., 2022) for the canonical code solutions.”
- The Stack: 3 TB of permissively licensed source code (2022, background): “Evaluation of code LLMs Code LLMs are frequently evaluated on text-to-code benchmarks, in which models are tasked with completing code snippets from natural language descriptions (Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Lai et al., 2022).”
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “Benchmarks like DS-1000 (Lai et al., 2023), ARCADE (Yin et al., 2022), NumpyEval (Zhang et al., 2023b), and PandasEval (Jain et al., 2022) focus on data science APIs.”
- NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness (2024, background): “…the community has been actively designing benchmarks (Hendrycks et al., 2021; Chen et al., 2021; Austin et al., 2021; Puri et al., 2021; Li et al., 2022; Liu et al., 2023b) with emphasis on generating code for a given problem specification of what functionality to achieve, e.g.,…”
- Measuring The Impact Of Programming Language Distribution (2023, background): “Existing datasets are either mono-lingual (Chen et al., 2021; Austin et al., 2021; Lai et al., 2022) or limited to only a subset of popular programming languages (Roziere et al.”
- Agentless: Demystifying LLM-based Software Engineering Agents (2024, methodology): “This is done by providing the previously described repository structure and asking the LLM to produce a list of irrelevant folders that do not need to be further inspected or modified to resolve the issue.”
- LEVER: Learning to Verify Language-to-Code Generation with Execution (2023, methodology): “A number of approaches were proposed to refine CodeLM sample selection, including test case execution (Li et al., 2022a), cross-sample similarity (Chen et al., 2021a; Li et al., 2022a; Shi et al., 2022) and maximum mutual information (Zhang et al., 2022) based filtering.”
- A Survey on Large Language Models for Code Generation (2024, unknown): “Firstly, the training and deployment of LLMs require immense processing power and memory.”
- xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2023, methodology): “(Chen et al., 2021) 1 1,325 MBPP (Austin et al., 2021) 1 1,500 TransCoder-ST (Roziere et al., 2021b) 3 ∞ APPS(Hendrycks et al., 2021) 1 22,711 MBXP (Athiwaratkun et al., 2022) 10 1,500 CodeContests (Li et al., 2022) 3 Our contributions in this paper are summarized as follows : 1.”
- CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “In addition, most of the related works (Hendrycks et al., 2021; Lai et al., 2023; Huang et al., 2022; Nijkamp et al., 2023; Chandel et al., 2022) focused on code generation tasks with target as Python when evaluating model performance, mainly because setting up a Python execution environment is…”
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, methodology): “Recently, open-domain coding benchmarks [58, 32] have been built to challenge code LLMs on application-specific scenarios by using specific sets of tools and libraries.”
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023, unknown): “, 2023) or design library-specific problems (Lai et al., 2022; Zan et al., 2022).”
- CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023, background): “…Metrics To alleviate previous issues, execution-based evaluation counts a generated code snippet as correct if it produces the required outputs when run with given inputs (Chen et al., 2021; Athiwaratkun et al., 2022; Li et al., 2022; Wang et al., 2022; Lai et al., 2022; Huang et al., 2022).”
- L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, methodology): “For other tasks, having a better pass@k also provides opportunities for post-generation reranking meth-ods such as (Shi et al., 2022; Zhang et al., 2023; Ni et al., 2023b).”
- A Survey of Large Language Models (2023, unknown): “Code Synthesis APPS [287], HumanEval [89], MBPP [140], CodeContest [97], MTPB [76], DS-1000 [288], ODEX [289]”
- A Comprehensive Overview of Large Language Models (2023, unknown): “Flan Muffin, T0-SF, NIv2, CoT MMLU [221], BBH [236], TyDiQA [261], MGSM [343] WizardCoder Code Alpaca HumanEval [246], MBPP [345], DS-1000 [349]”
- Aligning Large Language Models with Human: A Survey (2023, background): “The DS-1000 benchmark (Lai et al., 2022) comprises 1,000 distinct data science workflows spanning seven libraries.”
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct (2023, methodology): “Based on these results, our conclusion is that WizardCoder demonstrates a significant superiority over all other models when tackling data science problems on the DS-1000 benchmark.”