Transfer Learning in Natural Language Processing

By Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, Thomas Wolf. 2019

625 citations recorded by the source corpus.

The classic supervised machine learning paradigm is based on learning in isolation, a single predictive model for a task using a single dataset. This approach requires a large number of training examples and performs best for well-defined and narrow tasks. Transfer learning refers to a set of methods that extend this approach by leveraging data from additional domains or tasks to train a model with better generalization properties. Over the last two years, the field of Natural Language Processing (NLP) has witnessed the emergence of several transfer learning methods and architectures which significantly improved upon the state-of-the-art on a wide range of NLP tasks. These improvements together with the wide availability and ease of integration of these methods are reminiscent of the factors that led to the success of pretrained word embeddings and ImageNet pretraining in computer vision, and indicate that these methods will likely become a common tool in the NLP landscape as well as an important research direction. We will present an overview of modern transfer learning methods in NLP, how models are pre-trained, what information the representations they learn capture, and review examples and case studies on how these models can be integrated and adapted in downstream NLP tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “Current natural language processing (NLP) pipelines often make use of transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on a downstream task of interest (Ruder et al., 2019).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, positive): “…many Natural Language Processing (NLP) tasks have been dominated by deep learning models and, more speciﬁcally, the use of Transfer Learning methods (Ruder et al., 2019) in which a deep neural network language model is pretrained on a web-scale unlabelled text dataset with a general-purpose…”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “Transfer learning (Schmidhuber, 1987; Pratt et al., 1991; Caruana et al., 1995) has been the cornerstone of recent progress in natural language processing (Ruder et al., 2019; Devlin et al., 2019; Raffel et al., 2020).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “1 Introduction In the past 18 months, advances on many Natural Language Processing (NLP) tasks have been dominated by deep learning models and, more speciﬁcally, the use of Transfer Learning methods (Ruder et al., 2019) in which a deep neural network language model is pretrained on a web-scale unlab”
Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “…contain some long-range signal, which is required to perform long-range coreference resolution, reasoning with events, discourse understanding, etc. (Ruder et al., 2019) it seems to be overshadowed by the much stronger signal of short-term word co-occurrences and is thus difﬁcult to evaluate.”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Improve Model Robust Against Noise [167], [168], [169], [170], [171] Transfer Learning [172], [173], [174], [175], [176], [177], [178] 1340 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL.”
Continual Pre-Training Mitigates Forgetting in Language and Vision (2022, methodology): “Natural Language Processing (NLP), for example, often exploits Transfer Learning techniques (Ruder et al., 2019) implemented through the so-called pre-training fine-tuning setup.”
AdapterHub: A Framework for Adapting Transformers (2020, background): “As NLP tasks become more complex and require knowledge that is not directly accessible in a single monolithic pre-trained model (Ruder et al., 2019), adapters will provide NLP researchers and practitioners with many more sources of relevant information that can be easily combined in an efficient and”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “Transfer learning for NLP uses large pre-trained language models that are ﬁne-tuned on target tasks [Ruder et al., 2019, Devlin et al., 2019, Radford et al., 2019, Liu et al., 2019].”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “, 2017) model with a self-supervised loss and fine-tuning it on data of a downstream task (Ruder et al., 2019).”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, background): “Recent advances across a range of NLP tasks driven by large pre-trained language models (Wang et al., 2019; Ruder et al., 2019) has spurred further interest in this area as a way to probe their reasoning abilities.”
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2020, background): “Due to the limited amount of labelled training data available for many natural language processing (NLP) tasks, transfer learning has become ubiquitous [1].”
Want To Reduce Labeling Cost? GPT-3 Can Help (2021, methodology): “This work is complementary to many other methods based on human labeling, such as few-shot learning (Yin, 2020), active learning (Settles, 2009; Dor et al., 2020) and transfer learning (Ruder et al., 2019).”
Merging Models with Fisher-Weighted Averaging (2021, background): “Transfer learning has therefore become a particularly important and omnipresent tool across many ﬁelds, including natural language processing [57, 13, 9, 52, 53, 46] and computer vision [43, 24, 68].”
Efficient Intent Detection with Dual Sentence Encoders (2020, background): “training data to adapt to a particular task and/or domain (Ruder et al., 2019).”
Deep neural language modeling enables functional protein generation across families (2021, background): “These likelihoods showed in a significant drop in AUC of 0.”
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning (2021, methodology): “In generative and auto-regressive methods, next tokens are predicted given a causal context [10, 61] while denoising methods seek to reconstruct corrupted or masked tokens given an extended context [16, 50] For text, since the tokens (words or subwords [55, 67]) are discrete and has relatively high ”