From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

By Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, Aleksander Madry. 2020

151 citations recorded by the source corpus.

Building rich machine learning datasets in a scalable manner often necessitates a crowd-sourced data collection pipeline. In this work, we use human studies to investigate the consequences of employing such a pipeline, focusing on the popular ImageNet dataset. We study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset---including the introduction of biases that state-of-the-art models exploit. Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for. Finally, our findings emphasize the need to augment our current model training and evaluation toolkit to take such misalignments into account. To facilitate further research, we release our refined ImageNet annotations at this https URL.

Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

Citations in the corpus

Are we done with ImageNet? (2020, methodology): “Concurrently with our work, [26] proposes an improved pipeline for collecting ImageNet labels, although their analysis and conclusions differ from ours.”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “Tsipras et al. [2020] has revealed that the annotation pipeline for ImageNet does not reﬂect the intention of its development for the purpose of object recognition in images.”
Distilling Model Failures as Directions in Latent Space (2022, unknown): “For example, recent works (Tsipras et al., 2020; Vasudevan et al., 2022) need to resort to manual data exploration to identify label idiosyncrasies and failure modes in widely used datasets such as ImageNet.”
Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels (2021, background): “One of the most persistent and systematic types of label error on ImageNet is the erroneous single labels [46, 43, 53, 2], referring to the cases where only one out of multiple present”
Bugs in the Data: How ImageNet Misrepresents Biodiversity (2022, unknown): “There have been numerous studies of ImageNet from different angles, ranging from the quality of its labels (Beyer et al. 2020; Tsipras et al. 2020; Northcutt, Athalye, and Mueller 2021) to the biases that it reflects (Prabhu and Birhane 2021; Crawford and Paglen 2019; Shankar et al.”
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research (2021, methodology): “For example, the Fashion-MNIST dataset was introduced because 97 the original MNIST dataset came to be perceived as over-utilized and too easy [27], and the utility of 98 ImageNet — one of the most influential ML benchmarks in existence — as a meaningful measure 99 of progress has been a focus of cr”
It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks (2021, background): “ML studies have mainly explored the former through “benchmark misalignment” which arises with systemati-2 cally unstable labels [22, 23], but misalignment in the context of socio-technical systems is rarely discussed.”