Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research

By Bernard J. Koch, Emily L. Denton, A. Hanna, Jacob Gates Foster. 2021

180 citations recorded by the source corpus.

Benchmark datasets play a central role in the organization of machine learning research. They coordinate researchers around shared research problems and serve as a measure of progress towards shared goals. Despite the foundational role of benchmarking practices in this field, relatively little attention has been paid to the dynamics of benchmark dataset use and reuse, within or across machine learning subcommunities. In this paper, we dig into these dynamics. We study how dataset usage patterns differ across machine learning subcommunities and across time from 2015-2020. We find increasing concentration on fewer and fewer datasets within task communities, significant adoption of datasets from other tasks, and concentration across the field on datasets that have been introduced by researchers situated within a small number of elite institutions. Our results have implications for scientific evaluation, AI ethics, and equity/access within the field.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety Research

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “This re-use of datasets for incompatible tasks, along with the concentration on very few datasets, is a worrying trend that was quantified across multiple other tasks by Koch et al. (2021).”
Bugs in the Data: How ImageNet Misrepresents Biodiversity (2022, unknown): “Complementary work has been carried out to study how datasets are created and used by the community, finding that there is a concentration of power in terms of institutions that create, which results in influence of ML research and practice (Koch et al. 2021).”
Evaluation Gaps in Machine Learning Practice (2022, background): “However, ML research disciplines seem to be trending towards relying on fewer evaluation benchmark datasets [93], with test set reuse potentially leading to a research community’s overfitting with respect to the general task [103, 177].”
Advances, challenges and opportunities in creating data for trustworthy AI (2022, background): “Much of the recent research in AI has focused on improving model performance on several standard benchmark datasets such as ImageNet, CIFAR100 (Canadian Institute for Advanced Research, 100 classes), Adult Income, MIMIC (Medical Information Mart for Intensive Care) and so on 15–17 .”
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication (2021, background): “This has brought attention to the importance of conscientious data stewardship practices in all steps of the dataset life cycle, ranging from proper documentation at the data creation stage via checklists [58] and data sheets [34] to the study of dataset usage and genealogy [32, 50], licensing and a”