What makes ImageNet good for transfer learning?

By Minyoung Huh, Pulkit Agrawal, Alexei A. Efros. 2016

714 citations recorded by the source corpus.

The tremendous success of ImageNet-trained deep features on a wide range of transfer tasks begs the question: what are the properties of the ImageNet dataset that are critical for learning good, general-purpose features? This work provides an empirical investigation of various facets of this question: Is more pre-training data always better? How does feature quality depend on the number of training examples per class? Does adding more object classes improve performance? For the same data budget, how should the data be split into classes? Is fine-grained recognition necessary for learning good features? Given the same number of training classes, is it better to have coarse classes or fine-grained classes? Which is better: more classes or more examples per class? To answer these and related questions, we pre-trained CNN features on various subsets of the ImageNet dataset and evaluated transfer performance on PASCAL detection, PASCAL action classification, and SUN scene classification tasks. Our overall findings suggest that most changes in the choice of pre-training data long thought to be critical do not significantly affect transfer performance.? Given the same number of training classes, is it better to have coarse classes or fine-grained classes? Which is better: more classes or more examples per class?

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “…model pretraining and ﬁne-tuning and make them more scalable: for ImageNet, predicting far fewer classes only incurs a small performance drop (Huh et al., 2016), while recent work shows that an alignment be-tween source and target task label sets is important (Mahajan et al., 2018)—focusing…”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “computer vision [Oquab et al., 2014; Jia et al., 2014; Huh et al., 2016; Yosinski et al., 2014]) use a supervised dataset for pre-training, we were interested to see whether omitting the unsupervised task from the multi-task pre-training mixture still produced good results.”
Parameter-Efficient Transfer Learning for NLP (2019, background): “Transfer Learning in Visual Perception Fine-tuning models pre-trained on ImageNet (Deng et al., 2009) is ubiquitous when building image recognition models (Yosinski et al., 2014; Huh et al., 2016).”
Do Better ImageNet Models Transfer Better? (2018, background): “A substantial body of existing research indicates that, in image tasks, fine-tuning typically achieves higher accuracy than classification based on fixed features, especially for larger datasets or datasets with a larger domain mismatch from the training set [1, 6, 24, 74, 2, 43, 33, 9, 51].”
Fine-tuned Language Models for Text Classification (2018, background): “…direction is to improve the language model pretraining task and make it more scalable: for ImageNet, predicting far fewer classes only incurs a small performance drop (Huh et al., 2016)— focusing on predicting the most frequent words might retain most of the performance while speeding up training.”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, methodology): “We follow the experimental setup in [1, 15], which uses a nonstandard train/test split since it is computationally expensive to run all of our experiments on the 10 standard subsets proposed by [52].”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “Here, the goal is the development of representations — through either unsupervised or supervised means — that can generalize (with minimal ﬁne-tuning) to a wide range of other tasks they were not speciﬁcally developed for [e.g. Huh et al., 2016, Du et al., 2020, Bommasani et al., 2021].”
Reading Wikipedia to Answer Open-Domain Questions (2017, methodology): “Multitask learning (Caruana, 1998) and task transfer have a rich history in machine learning (e.g., using ImageNet in the computer vision community (Huh et al., 2016)), as well as in NLP in particular (Collobert and Weston, 2008).”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “This pre-training is often referred to as representation learning; representations can be trained using supervised [32, 36], weakly-supervised [44], or unsupervised data [13, 14, 66, 70].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “trained on ImageNet labels is a popular strategy for transferring representations to new tasks [26].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “Indeed, the most popular models used in practice are pre-trained on ImageNet labels (Huh et al., 2016, and refs therein).”
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, methodology): “[19] showed that using a smaller subset of images for training from ImageNet hurts performance.”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “[33] in order to allow for a direct comparison with their method.”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “Self-supervision builds on the fact that convolutional neural networks (CNNs) transfer well between tasks (Shin et al., 2016; Oquab et al., 2014; Girshick, 2015; Huh et al., 2016).”
Scattering Networks for Hybrid Representation Learning (2018, methodology): “When trained on large-scale datasets they have been shown to produce representations that are transferable to other datasets [25], [59], which indicates they have captured generic properties of a”
ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, methodology): “Experiments It has been shown that pretraining the model on a large-scale dataset, such as MS COCO [35], improves generalization and helps to prevent overﬁtting [23, 26, 52].”
Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, background): “[37] have analyzed the factors that contributed to ImageNet’s wide adoption.”
Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, background): “However, given that data from a different distribution can be beneﬁcial for a given task (Huh et al., 2016), it is conceivable that the need for task-speciﬁc data could be obviated with pre-training.”
Few-Shot Image Recognition by Predicting Parameters from Activations (2017, background): “More importantly, the semantic structures [12] are also preserved in both activations and parameters, indicating a promising generalizability to unseen categories.”