Do Better ImageNet Models Transfer Better?

By Simon Kornblith, Jonathon Shlens, Quoc V. Le. 2018

1,561 citations recorded by the source corpus.

Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer. An implicit hypothesis in modern computer vision research is that models that perform better on ImageNet necessarily perform better on other vision tasks. However, this hypothesis has never been systematically tested. Here, we compare the performance of 16 classification networks on 12 image classification datasets. We find that, when networks are used as fixed feature extractors or fine-tuned, there is a strong correlation between ImageNet accuracy and transfer accuracy (r = 0.99 and 0.96, respectively). In the former setting, we find that this relationship is very sensitive to the way in which networks are trained on ImageNet; many common forms of regularization slightly improve ImageNet accuracy but yield features that are much worse for transfer learning. Additionally, we find that, on two small fine-grained image classification datasets, pretraining on ImageNet provides minimal benefits, indicating the learned features from ImageNet do not transfer well to fine-grained tasks. Together, our results show that ImageNet architectures generalize well across datasets, but ImageNet features are less general than previously suggested.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

In Evaluation & Benchmarks

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “There is some early empirical work along these lines in the field of computer vision [Huh et al., 2016; Kornblith et al., 2018; He et al., 2018].”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “None of these tasks are measured in the evaluation suite of Kornblith et al. (2019).”
A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Following Kornblith et al. (2019), we perform hyperparameter tuning for each model-dataset combination and select the best hyperparameters on a validation set.”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, methodology): “In CV, generally outperforms when transferring from ImageNet supervised classiﬁcation pretraining to other classiﬁcation tasks (Kornblith et al., 2018).”
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “Transfer accuracy and #params for NASNet (Zoph et al., 2018), Inception-v4 (Szegedy et al., 2017), ResNet-152 (He et al., 2016) are from (Kornblith et al., 2019). weight decay 1e-5; initial learning rate 0.256 that decays by 0.97 every 2.4 epochs.”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “It has been observed that better IMAGENET models transfer better [37, 72].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “[34] show that better supervised models tend to transfer better when fine-tuned.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “We conduct experiments on ImageNet 2012 ILSVRC challenge prediction task since it has been considered one of the most heavily benchmarked datasets in computer vision and that improvements on ImageNet transfer to other datasets [34, 55].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “Here, we ﬁne-tuning the entire network on the task data, since that performs best (Kornblith et al., 2019).”
Fixing the train-test resolution discrepancy (2019, background): “Furthermore, advances in image classification translate to improved results on many other tasks [12, 18].”
Attention Augmented Convolutional Networks (2019, methodology): “These datasets have been used as benchmarks for delineating better image featurizations and network architectures across a broad range of tasks [21].”
Learning Visual Representations with Caption Annotations (2020, background): “[9] shows that transfer learning performance is correlated with the overlap of classes between IN-1K and target task datasets.”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, methodology): “The research community is reasonable in judging IID accuracy with a univariate metric like ImageNet classification accuracy, as models with higher ImageNet accuracy reliably have better fine-tuned classification accuracy on other tasks (Kornblith et al., 2018).”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “We perform linear evaluation and fine-tuning on the same set of classification tasks used in [8, 74], and carefully follow their evaluation protocol, as detailed in Appendix E.”
Projected GANs Converge Faster (2021, neutral): “Using pretrained representations has become ubiquitous in computer vision [29, 30, 48]”
Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, methodology): “In particular, we apply SAM to ﬁnetuning EfﬁcentNet-b7 (pretrained on ImageNet) and EfﬁcientNet-L2 (pretrained on ImageNet plus unlabeled JFT; input resolution 475) (Tan & Le, 2019; Kornblith et al., 2018; Huang et al., 2018).”