DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation

By Yuhang Lai, Chengxi Li, Tianyi Zhang, Ruiqi Zhong, Luke S. Zettlemoyer, Scott Yih, Daniel Fried, Si-yi Wang, Tao Yu. 2022

563 citations recorded by the source corpus.

We introduce DS-1000, a code generation benchmark with a thousand data science problems spanning seven Python libraries, such as NumPy and Pandas. Compared to prior works, DS-1000 incorporates three core features. First, our problems reflect diverse, realistic, and practical use cases since we collected them from StackOverflow. Second, our automatic evaluation is highly specific (reliable) -- across all Codex-002-predicted solutions that our evaluation accept, only 1.8% of them are incorrect; we achieve this with multi-criteria metrics, checking both functional correctness by running test cases and surface-form constraints by restricting API usages or keywords. Finally, we proactively defend against memorization by slightly modifying our problems to be different from the original StackOverflow source; consequently, models cannot answer them correctly by memorizing the solutions from pre-training. The current best public system (Codex-002) achieves 43.3% accuracy, leaving ample room for improvement. We release our benchmark at https://ds1000-code-gen.github.io.

Subfields: Artificial Intelligence, Information Systems, Materials Chemistry, Evaluation & Benchmarks, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Graham Neubig
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Mohit Bansal
  5. Percy Liang
  6. Caiming Xiong
  7. Yoshua Bengio
  8. Noah A. Smith

In Information Systems

  1. Chun Xia
  2. Lingming Zhang
  3. Uri Alon
  4. Graham Neubig
  5. Yinlin Deng
  6. Charles Sutton
  7. Koushik Sen
  8. Shuyan Zhou

In Materials Chemistry

  1. Siva Reddy
  2. Graham Neubig
  3. Caiming Xiong
  4. Ziyang Luo
  5. Oleksandr Polozov
  6. Shafiq R. Joty
  7. Charles Sutton
  8. Uri Alon

In Evaluation & Benchmarks

  1. Graham Neubig
  2. Dragomir R. Radev
  3. Shafiq R. Joty
  4. Chun Xia
  5. Lingming Zhang
  6. Caiming Xiong
  7. Marc Marone
  8. Shuyan Zhou

In Code Generation & Program Synthesis

  1. Graham Neubig
  2. Charles Sutton
  3. Uri Alon
  4. Shuyan Zhou
  5. Zhiruo Wang
  6. Koushik Sen
  7. Qian Liu
  8. Wen-tau Yih

Citations in the corpus