How transferable are features in deep neural networks?
By Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson. 2014
9,349 citations recorded by the source corpus.
Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Sergey Levine
- Jianfeng Gao
- Percy Liang
- Dacheng Tao
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Bernt Schiele
- Andrew Zisserman
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Trevor Darrell
- Dacheng Tao
- Andrea Vedaldi
- Mingsheng Long
- Jianmin Wang
- Jiashi Feng
- Ishan Misra
Citations in the corpus
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “Computer vision research has also demonstrated the importance of transfer learning from large pre-trained models, where an effective recipe is to fine-tune models pre-trained with ImageNet (Deng et al., 2009; Yosinski et al., 2014).”
- Universal Language Model Fine-tuning for Text Classification (2018, background): “Discriminative fine-tuning As different layers capture different types of information ( Yosinski et al. , 2014), they should be fine-tuned to different extents .”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “computer vision [Oquab et al., 2014; Jia et al., 2014; Huh et al., 2016; Yosinski et al., 2014]) use a supervised dataset for pre-training, we were interested to see whether omitting the unsupervised task from the multi-task pre-training mixture still produced good results.”
- Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “A variety of approaches have been proposed, including re-training the model in the target domain (e.g., [52]), adapting the weights of the model based on the statistics of the source and target domains (e.g., [22]), learning invariant features between domains (e.g., [47]), and learning a mapping…”
- To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “1 In computer vision (CV), Yosinski et al. (2014) similarly found that the transferability of 1 Mou et al. (2016), however, only investigate transfer be-tween classification tasks (NLI → SICK-E/MRPC). features decreases as the distance between the pre-training and target task increases.”
- Parameter-Efficient Transfer Learning for NLP (2019, background): “Transfer Learning in Visual Perception Fine-tuning models pre-trained on ImageNet (Deng et al., 2009) is ubiquitous when building image recognition models (Yosinski et al., 2014; Huh et al., 2016).”
- Do Better ImageNet Models Transfer Better? (2018, background): “A substantial body of existing research indicates that, in image tasks, fine-tuning typically achieves higher accuracy than classification based on fixed features, especially for larger datasets or datasets with a larger domain mismatch from the training set [1, 6, 24, 74, 2, 43, 33, 9, 51].”
- What makes ImageNet good for transfer learning? (2016, background): “Factors that affect fine-tuning: The question of whether pre-training should be terminated early to prevent overfitting and what layers should be used for transfer learning was studied by [2, 49].”
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “When the upstream and downstream datasets differ significantly, fine-tuning the original weights is more effective (Yosinski et al., 2014; Ko-rnblith et al., 2019).”
- Learning Features by Watching Objects Move (2016, methodology): “We also down-sample the ground truth mask to s × s and sum the cross entropy losses over the s 2 locations to train the network.”
- GSNs : Generative Stochastic Networks (2015, background): “…supervised models raises the question: can we take advantage of this progress to improve our ability to train deep, generative, unsupervised, semisupervised (Goodfellow et al., 2015), transfer learning (Galanti et al., 2016; Yosinski et al., 2014) or structured output (Li et al., 2013) models?”
- Predicting multicellular function through multi-layer tissue networks (2017, positive): “Generally speaking, we observed that the transferability of classifiers decreased when the tree-based distance between the source and the target tissue in the tissue hierarchy increased, which is consistent with the empirical evidence in transfer learning (Yosinski et al. , 2014).”
- A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “We are motivated by the fact that the first few layers of most networks extract low-level information (Yosinski et al., 2014), and thus learning them may not require the high-level semantic information captured by manual labels.”