dcbench: a benchmark for data-centric AI systems
By Sabri Eyuboglu, Bojan Karlas, Christopher R'e, Ce Zhang, James Y. Zou. 2022
23 citations recorded by the source corpus.
The development workflow for today's AI applications has grown far beyond the standard model training task. This workflow typically consists of various data and model management tasks. It includes a "data cycle" aimed at producing high-quality training data, and a "model cycle" aimed at managing trained models on their way to production. This broadened workflow has opened a space for already emerging tools and systems for AI development. However, as a research community, we are still missing standardized ways to evaluate these tools and systems. In a humble effort to get this wheel turning, we developed dcbench, a benchmark for evaluating systems for data-centric AI development. In this report, we present the main ideas behind dcbench, some benchmark tasks that we included in the initial release, and a short summary of its implementation.
Subfields: Artificial Intelligence, Information Systems and Management, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Hannaneh Hajishirzi
- Percy Liang
- Yejin Choi
- Luke S. Zettlemoyer
- Douwe Kiela
- Ali Farhadi
- Noah A. Smith
In Information Systems and Management
In Evaluation & Benchmarks
- Ali Farhadi
- Ludwig Schmidt
- Yair Carmon
- Vaishaal Shankar
- Tristan Thrush
- Mitchell Wortsman
- Gabriel Ilharco
- Douwe Kiela
Citations in the corpus
- Measuring Data (2022, background): “, 2021), benchmarking (Eyuboglu et al., 2022), and debugging (Rajani et al.”
- DataComp: In search of the next generation of multimodal datasets (2023, background): “[39] Sabri Eyuboglu, Bojan Karlaš, Christopher Ré, Ce Zhang, and James Zou.”