Are we done with ImageNet?

By Lucas Beyer, Olivier J. H'enaff, Alexander Kolesnikov, Xiaohua Zhai, Aäron van den Oord. 2020

483 citations recorded by the source corpus.

Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “We transfer the models trained on these dataset to several benchmark tasks: ImageNet on the original validation labels and the cleaned-up ReaL labels (Beyer et al., 2020), CIFAR-10/100 (Krizhevsky, 2009), Oxford-IIIT Pets (Parkhi et al., 2012), and Oxford Flowers-102 (Nilsback & Zisserman, 2008).”
Training data-efficient image transformers & distillation through attention (2020, methodology): “image throughput Imagenet [39] Real [5] V2 [38] size (image/s) acc.”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Only on ReaL and ObjectNet does there exist an individual model that performs statistically signiﬁcantly better than the soup, and the best model differs between those two datasets.”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “We evaluate robustness on ImageNetv2 [48], -R [26, 63], -A [27], -ReaL [2], and ObjectNet [1], following CLIP and ALIGN.”
Scaling Vision Transformers (2021, methodology): “Since we plot error rate, the information-theoretic interpretation does not apply, but our observations support the notion of fundamental performance ceilings for ImageNet [4].”
FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, methodology): “The importance of reliable evaluation sets is highlighted by recent research in computer vision which focuses on improving the evaluation and/or validation sets of widely-used datasets— [44] for CIFAR-10/-100 [45]; and [46, 47] for ImageNet [48].”
Large image datasets: A pyrrhic win for computer vision? (2020, neutral): “Within the topic of image labeling but with a different dimension and focus, recent work such as [80] and [6] has focused on the shortcomings of human-annotation procedures used during the ImageNet dataset curation.”
MLP-Mixer: An all-MLP Architecture for Vision (2021, methodology): ““ImNet” and “ReaL” columns refer to the original ImageNet validation [13] and cleaned-up ReaL [5] [42] and 86.5% NFNet [7].”
Going deeper with Image Transformers (2021, methodology): “• Our best CaiT models establish the new state of the art on Imagenet-Real [6] and Imagenet V2 matched frequency [53] with no additional training data.”
CvT: Introducing Convolutions to Vision Transformers (2021, methodology): “We compare our method with state-of-the-art classiﬁcation methods including Transformer-based models and representative CNN-based models on ImageNet [9], ImageNet Real [2] and ImageNet V2 [26] datasets in Table 3.”
Meta Pseudo Labels (2020, methodology): “Table 1: Summary of our key results on ImageNet ILSVRC 2012 validation set [56] and the ImageNet-ReaL test set [6].”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “, 2019), ReaL (Beyer et al., 2020) – Image classification (zero-shot): ImageNet (Deng et al.”
Incorporating Convolution Designs into Visual Transformers (2021, methodology): “We report the results on ImageNet validation dataset and ImageNet Real dataset [4] in Table 4.”
How Well Do Sparse ImageNet Models Transfer? (2021, methodology): “Namely, we use the ImageNet “re-assessed labels” [2], where the original ImageNet validation images are re-assessed by human annotators.”