A Comprehensive Survey on Transfer Learning

By Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, Qing He. 2019

5,955 citations recorded by the source corpus.

Transfer learning aims at improving the performance of target learners on target domains by transferring the knowledge contained in different but related source domains. In this way, the dependence on a large number of target-domain data can be reduced for constructing target learners. Due to the wide application prospects, transfer learning has become a popular and promising area in machine learning. Although there are already some valuable and impressive surveys on transfer learning, these surveys introduce approaches in a relatively isolated way and lack the recent advances in transfer learning. Due to the rapid expansion of the transfer learning area, it is both necessary and challenging to comprehensively review the relevant studies. This survey attempts to connect and systematize the existing transfer learning research studies, as well as to summarize and interpret the mechanisms and the strategies of transfer learning in a comprehensive way, which may help readers have a better understanding of the current research status and ideas. Unlike previous surveys, this survey article reviews more than 40 representative transfer learning approaches, especially homogeneous transfer learning approaches, from the perspectives of data and model. The applications of transfer learning are also briefly introduced. In order to show the performance of different transfer learning models, over 20 representative transfer learning models are used for experiments. The models are performed on three different data sets, that is, Amazon Reviews, Reuters-21578, and Office-31, and the experimental results demonstrate the importance of selecting appropriate transfer learning models for different applications in practice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “Transfer learning methods are implemented to learn the high-level ConvNet-based representations in limited information content [47,51].”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “However, training Cφ via standard supervised training on all generated texts is likely to yield suboptimal performance on downstream tasks because (1) the generated texts may contain noise asGθ may not always produce texts pertaining to the desired class, especially for challenging sequence pair tas”
Visual Prompt Tuning (2022, background): “One straightforward approach is to turn to other strategies that we have per-fected for adapting ConvNets to new tasks, as in Fig.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “Transfer learning from generic pre-trained models has enabled signiﬁcant recent progress in ML (Zhuang et al., 2021).”
Knowledge Capture and Replay for Continual Learning (2020, background): “Approaches such as transfer learning [25] and knowledge distillation [7] enable representational or functional translation [9] from one model to the other.”
A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU (2023, background): “For example, someone who has learned to play the violin is likely to learn the piano more quickly due to the shared characteristics between musical instruments [100].”
A Decade Survey of Transfer Learning (2010–2020) (2020, background): “More recently, another survey [5] gives special attention to homogeneous TL and reviews of state-of-the-art homogeneous TL algorithms and applications.”
Smarter eco-cities and their leading-edge artificial intelligence of things solutions for environmental sustainability: A comprehensive systematic review (2023, background): “These add to transduction learning, multitasking learning [73], federated learning [74], transfer learning [75], and few-shot learning [76].”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, background): “An altogether different approach may leverage recent advances in transfer learning focused on understanding task relationships [54, 3, 15, 58, 2]; however, [46] show transfer learning algorithms which determine task similarity do not carry over to the multi-task learning domain and instead propose…”
Deep Transfer Learning for Bearing Fault Diagnosis: A Systematic Review Since 2016 (2023, background): “[56] surveyed more than 40 homogeneous transfer learning methods to give readers a comprehensive”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “Similarly, transfer learning focuses on improving models that would generalize well on other domains (distributions), whereas our work investigates how examples in different distributions affect human performance [36, 62, 74].”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “2) Transfer Learning and Domain Adaptation: Transfer learning is a sub-ﬁeld of machine learning, aiming to address the learning problem of a target task without sufﬁcient training data by transferring the learned knowledge from a source related task [198].”
Domain Adaptation for Medical Image Analysis: A Survey (2021, neutral): “The learning model is only allowed to be trained on several related source domains to gain enough generalization ability for the target domain.”
Structural Autoencoders Improve Representations for Generation and Transfer (2020, background): “However, performing well on a highly speciﬁc dataset is usually insufﬁcient to satisfactorily solve real-world problems [1, 2].”
A Survey on Deep Semi-Supervised Learning (2021, background): “Transfer learning [37], [38], [39] aims to apply knowledge from one or more source domains to a target domain in order to improve performance on the target task.”
Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey (2020, positive): “DL yields good results when larger volumes of data are available, but in the case of transfer learning, the same can be achieved with limited labeled dataset (Zhuang et al., 2020).”
Review of multi-fidelity models (2016, background): “As the field evolves, researchers are keenly exploring methods like transfer learning, fine-tuning and ensemble techniques to update these models efficiently without compromising their existing knowledge [245, 205, 12].”
Transfer learning for medical image classification: a literature review (2022, background): “[8] reviewed more than forty representative TL approaches from the perspectives of data and models.”