Neural transfer learning for natural language processing

By Sebastian Ruder. 2019

251 citations recorded by the source corpus.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “Sequential inductive transfer learning (Pan and Yang, 2010; Ruder, 2019) consists of two stages: pretraining, in which the model learns a generalpurpose representation of inputs, and adaptation, in which the representation is transferred to a new task.”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, methodology): “While transfer learning has become the go-to method for solving NLP tasks (Pan and Yang, 2010; Torrey and Shavlik, 2010; Ruder, 2019; Howard and Ruder, 2018; Peters et al., 2018), transformerbased models are notoriously deep requiring millions or even billions of parameters (Radford et al.”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “In this section, we formalize our goal of transfer learning (Pan and Yang, 2010; Torrey and Shavlik, 2010; Ruder, 2019), highlight its key challenges, and provide a brief overview of common methods that can be used to address them.”
Hierarchical Inductive Transfer for Continual Dialogue Learning (2022, background): “Inductive Transfer Continual learning in terms of transferring inductive knowledge from pretrained models to downstream tasks can be categorized into feature-based, fine-tuning–based, and adapter-based (Ruder, 2019).”
Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research (2019, background): “A different way in which models can improve over others is in how well they are suited for transfer to other tasks (see Ruder (2019) for a recent overview of such transfer-learning).”
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020, background): “DA is a particular case of transfer learning, namely transductive transfer learning (Pan and Yang, 2009; Ruder, 2019).”
KinGDOM: Knowledge-Guided DOMain Adaptation for Sentiment Analysis (2020, background): “focus is on the third category which has emerged as a popular approach in this deep representation learning era (Ruder, 2019; Poria et al., 2020).”
Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks (2019, unknown): “We refer the reader to Ruder (2019) for a more thorough discussion of transfer learning.”
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, background): “These successes are mainly achieved through Sequential Transfer Learning (Ruder, 2019): pretrain a language model on large-scale unlabeled data and then adapt it to downstream tasks.”
A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning (2019, methodology): “This motivates the use of pre-training and transfer learning, which has shown to be highly effective in many natural language processing (NLP) scenarios in which there is not enough annotated data, notably in low-resource machine translation (MT) (Lample et al., 2018b; Ruder, 2019).”
An Analysis of Simple Data Augmentation for Named Entity Recognition (2020, background): “For example, transfer learning pretrains language representations on self-supervised or rich-resource source tasks and then adapts these representations to the target task (Ruder, 2019; Gururangan et al., 2020).”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, methodology): “To ease off-policy learning, we propose to add a KL-regularization term in the RL loss (Jaques et al., 2017, 2019; Wu et al., 2019), and refer to it as TrufLL off , KL .”