A Survey on Transfer Learning
By Sinno Jialin Pan, Qiang Yang. 2010
23,590 citations recorded by the source corpus.
A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. However, in many real-world applications, this assumption may not hold. For example, we sometimes have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a different feature space or follow a different data distribution. In such cases, knowledge transfer, if done successfully, would greatly improve the performance of learning by avoiding much expensive data-labeling efforts. In recent years, transfer learning has emerged as a new learning framework to address this problem. This survey focuses on categorizing and reviewing the current progress on transfer learning for classification, regression, and clustering problems. In this survey, we discuss the relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift. We also explore some potential future issues in transfer learning research.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Sergey Levine
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
- Dacheng Tao
- Trevor Darrell
In Representation & Self-Supervised Learning
- Mingsheng Long
- Jianmin Wang
- Trevor Darrell
- Dacheng Tao
- Mohammad Rostami
- Kate Saenko
- Andrea Vedaldi
- Qiang Yang
Citations in the corpus
- Universal Language Model Fine-tuning for Text Classification (2018, background): “We are interested in the most general inductive transfer learning setting for NLP (Pan and Yang, 2010): Given a static source task T S and any target task T T with T S 6 = T T , we would like to improve performance on T T .”
- Story Ending Prediction by Transferable BERT (2019, methodology): “Inspired by transfer learning techniques [ Pan and Yang, 2009 ] , we consider incorporating supervised knowledge into this conventional pre-training framework to find a better initialization for the target task.”
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…closely related to multitask learning include zero or few-shot learning (Lake et al., 2011; Romera-Paredes and Torr, 2015; Vinyals et al., 2016; Pan and Yang, 2010), meta-learning and life-long learning (Thrun and Mitchell, 1995; Silver et al., 2013; Chen and Liu, 2016; Parisi et al., 2018;…”
- Transfer Learning in Natural Language Processing (2019, methodology): “The tutorial will start with a broad overview of transfer learning methods following Pan and Yang (2010).”
- To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “Sequential inductive transfer learning (Pan and Yang, 2010) consists of two stages: pretraining , in which the model learns a general-purpose representation of inputs, and adaptation , in which the representation is transferred to a new task.”
- An Overview of Multi-task Learning (2018, background): “MTL is related to other areas in machine learning, including transfer learning [2],multi-label learning [3] andmulti-output regression, but exhibits different characteristics.”
- Fine-tuned Language Models for Text Classification (2018, methodology): “ng, we will show that fine-tuning is superior to both hypercolumns and MTL as a general-purpose transfer learning method for NLP. 3 Transfer Learning for NLP Transfer learning has many instantiations (Pan and Yang, 2010). Here, we are interested in the most general inductive transfer learning setting”
- Snorkel: Rapid Training Data Creation with Weak Supervision (2017, background): “Transfer learning considers the strategy of repurposing models trained on different datasets or tasks where labeled training data is more abundant [34].”
- On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “Transferability of image classification models In transfer learning [48], a model might leverage the data it has seen on a related distribution, Ppre−train, to perform better on a new task Ptrain.”
- Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “A network is trained once on a large, generic dataset, and its weights are then used to initialize subsequent tasks which can be solved with fewer data points, and less compute [40,44,14].”
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Improving sample efficiency has been approached from many angles: few-shot learning (Fei-Fei et al., 2006), transfer learning (Pan & Yang, 2009), domain adaptation (Wang & Deng, 2018), and representation learning (Bengio et al., 2013).”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “A complementary approach uses data from the target domain in order to improve generalization on that particular domain [64].”
- Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “For a general primer, we refer the reader to the following reviews (Weiss et al., 2016; Csurka, 2017; Wang and Deng, 2018).”
- Refining Language Models with Compositional Explanations (2021, background): “However, when a source model (finetuned on some upstream dataset) is applied to a target domain with a different data distribution, the model may suffer from poor performance due to some spurious feature patterns learned from the upstream dataset [33, 44, 6].”
- AdapterDrop: On the Efficiency of Adapters in Transformers (2020, methodology): “While transfer learning has become the go-to method for solving NLP tasks (Pan and Yang, 2010; Torrey and Shavlik, 2010; Ruder, 2019; Howard and Ruder, 2018; Peters et al., 2018), transformer-based models are notoriously deep requiring millions or even billions of parameters (Radford et al., 2018;…”
- AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “In this section, we formalize our goal of transfer learning (Pan and Yang, 2010; Torrey and Shavlik, 2010; Ruder, 2019), highlight its key challenges, and provide a brief overview of common methods that can be used to address them.”
- Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines (2018, background): “The input distribution (domain) difference has been discussed extensively in the setting of transfer learning [15], primarily as a domain adaptation problem [16].”
- Training Neural Networks with Fixed Sparse Masks (2021, background): “Transfer learning [36], where a model is initialized from a pre-trained checkpoint before being fine-tuned on a related downstream task, can dramatically improve performance and speed up convergence on the downstream task [12, 8, 40].”
- LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, background): “Transfer learning [44] has been a successful paradigm in computer vision [32,33,42].”
- Billion-scale semi-supervised learning for image classification (2019, background): “This is a historical line of research in machine learning, see the survey [30] by Pan and Yang for an extensive classification of the different problems and approaches related to this topic.”