Evaluating Machine Accuracy on ImageNet

By Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, B. Recht, Ludwig Schmidt. 2020

158 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Indeed, recent work experimentally conﬁrms that humans achieve similar classiﬁcation accuracy on the original ImageNet test set and the ImageNetV2 replication study (one of the distribution shifts in our testbed) [77].”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “In Table 4, we report results on the ImageNet validation set and the five distribution shift datasets studied above as well as two relabeled ImageNet validation sets, ReaL [10] and multilabel [79].”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “In [51], ImageNet dataset evaluation was revisited and alternative test sets were released.”
Robust fine-tuning of zero-shot models (2021, background): “Figure 8: WiSE-FT improves OOD accuracy while maintaining or improving ID accuracy on ImageNet-Vid-Robust, YTBB-Robust (Shankar et al., 2019), CIFAR-10.”
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, background): “Whereas train set labels in a small number of machine learning datasets, e.g. in the ImageNet dataset, are well-known to contain errors [16, 33, 40], labeled data in test sets is often considered “correct” as long as it is drawn from the same distribution as the train set.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “However, it remains unclear how much such errors impact performance measurement, if at all, especially for deep learning [139].”
Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization (2021, result): “…the fact that this trend is still below x “ y suggests that the drop is also partially due to CIFAR-10 reproductions being harder than CIFAR-10 for current methods (interestingly, humans show similar performance on both test sets (Recht et al., 2019; Miller et al., 2020; Shankar et al., 2020)).”
Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels (2021, methodology): “It took more than three months for five human experts (authors of [39]) to label mere 2,000 images.”