Do ImageNet Classifiers Generalize to ImageNet?

By B. Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar. 2019

2,251 citations recorded by the source corpus.

We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “ImageNetV2 closely followed the creation process of the original ImageNet dataset which suggests that gains in accuracy from supervised adaptation are closely concentrated around the ImageNet distribution.”
Understanding Dataset Difficulty with V-Usable Information (2021, positive): “Datasets are designed to act as proxies for real-world tasks, yet most bear limited semblance to the tasks they purport to reflect (Torralba & Efros, 2011; Recht et al., 2019).”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “While existing work shows that we do not overfit to IMAGENET, it is possible that these models have correlated failure modes on datasets which share the biases with IMAGENET [52].”
Are we done with ImageNet? (2020, background): “The most related recent work [23] attempts to replicate the ImageNet collection pipeline, which results in a 12% drop in accuracy.”
Fixing the train-test resolution discrepancy: FixEfficientNet (2020, methodology): “The original study of [17] shows that there is significant overfitting of various models to the Imagenet 2012 valuation set, but that it does not impact the relative order of the models.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “[81, 31, 66, 27], have shown that vision models lack robustness.”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Fortunately, despite numerous test re-evaluations on popular ML benchmarks, such as ImageNet, progress seems to transfer to new data (Recht et al., 2018; 2019; Kornblith et al., 2019).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “Our testbed consists of 109 model predictions and is 100 times larger than prior work [27, 33, 47, 68].”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, methodology): “Recent works [12, 26, 14] have begun to characterize model performance on out-ofdistribution (OOD) data with various new test sets, with dissonant findings.”
The Effect of Natural Distribution Shift on Question Answering Models (2020, methodology): “, 2019), and ImageNet (Recht et al., 2019).”
Natural Adversarial Examples (2019, methodology): “[37] remind us that ImageNet test examples tend to be relatively simple close-up images, so that the current test set may be too easy and not represent harder images encountered in the real world.”
Do Image Classifiers Generalize Across Time? (2019, methodology): “As an example, photo sharing websites contain a large number of near-duplicate images: pairs of images of the same scene captured at different times, viewpoints, or from a different camera [25].”
Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “The independent collection procedure is more similar to (Recht et al., 2019), who collected a new set of standard colorful ImageNet validation images.”
Early Convolutions Help Transformers See Better (2021, background): “Additional results on the ImageNet-V2 [33] test set are presented in the Appendix.”
VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “Our models perform even better on the ImageNet-V2 benchmark.”
Training data-efficient image transformers & distillation through attention (2020, methodology): “Table 6: Throughput on and accuracy on Imagenet [39], Imagenet Real [5] and Imagenet V2 matched frequency [38] of DeiT and of several state-of-the-art convnets, for models trained with no external data.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Training a machine learning model with a learned data augmentation policy may significantly improve image classification [4, 19, 15], object detection [51], model robustness [25, 46, 34], and semi-supervised learning image classification [44].”