Bugs in the Data: How ImageNet Misrepresents Biodiversity
By Alexandra Sasha Luccioni, David Rolnick. 2022
52 citations recorded by the source corpus.
ImageNet-1k is a dataset often used for benchmarking machine learning (ML) models and evaluating tasks such as image recognition and object detection. Wild animals make up 27% of ImageNet-1k but, unlike classes representing people and objects, these data have not been closely scrutinized. In the current paper, we analyze the 13,450 images from 269 classes that represent wild animals in the ImageNet-1k validation set, with the participation of expert ecologists. We find that many of the classes are ill-defined or overlapping, and that 12% of the images are incorrectly labeled, with some classes having >90% of images incorrect. We also find that both the wildlife-related labels and images included in ImageNet-1k present significant geographical and cultural biases, as well as ambiguities such as artificial animals, multiple species in the same image, or the presence of humans. Our findings highlight serious issues with the extensive use of this dataset for evaluating ML systems, the use of such algorithms in wildlife-related tasks, and more broadly the ways in which ML datasets are commonly created and curated.
Subfields: Computer Vision and Pattern Recognition, Ecological Modeling, Ecology, Molecular Biology, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Douwe Kiela
- Nazneen Rajani
- Tristan Thrush
- Angelina McMillan-Major
- Yacine Jernite
- Alexandra Sasha Luccioni
- Margaret Mitchell
- Mohit Bansal
In Ecological Modeling
In Molecular Biology
In Evaluation & Benchmarks
- Tristan Thrush
- Douwe Kiela
- Angelina McMillan-Major
- Yacine Jernite
- Alexandra Sasha Luccioni
- Margaret Mitchell
- Nazneen Rajani
- Mohit Bansal
Citations in the corpus
- Measuring Data (2022, methodology): “Fit to “Zipf’s Law” (Meister and Cotterell, 2021a; Luccioni et al., 2022) (language ): A familiar quantity in corpus linguistics and NLP, measurements based on Zipf’s law quantify how closely the quantities of an item in a dataset or data batch match a Zipfian distribution, which is an inverse…”