Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?

By Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman. 2020

210 citations recorded by the source corpus.

While pretrained models such as BERT have shown large gains across natural language understanding tasks, their performance can be improved by further training the model on a data-rich intermediate task, before fine-tuning it on a target task. However, it is still poorly understood when and why intermediate-task training is beneficial for a given target task. To investigate this, we perform a large-scale study on the pretrained RoBERTa model with 110 intermediate-target task combinations. We further evaluate all trained models with 25 probing tasks meant to reveal the specific skills that drive transfer. We observe that intermediate tasks requiring high-level inference and reasoning abilities tend to work best. We also observe that target task performance is strongly correlated with higher-level abilities such as coreference resolution. However, we fail to observe more granular correlations between probing and target task performance, highlighting the need for further work on broad-coverage probing benchmarks. We also observe evidence that the forgetting of knowledge learned during pretraining may limit our analysis, highlighting the need for further work on transfer learning methods in these settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “While prior work has shown the benefits of intermediate finetuning (Pruksachatkun et al., 2020; Vu et al., 2020) or multi-task finetuning (Aghajanyan et al.”
BEiT: BERT Pre-Training of Image Transformers (2021, background): “[DBK+20] also report that pixel-level auto-encoding performs worse, although it is the most straightforward translation of BERT from NLP to CV.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “For Glue-COLA, similar observa- tions are reported by Pruksachatkun et al. (2020) in an intermediate-task transfer learning setting, where the authors conjecture catastrophic forget- ting of the masked language modeling (MLM) tasks may be the cause.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “, 2019), successful transfer (He et al., 2019; Pruksachatkun et al., 2020), and out-of-distribution robustness (Hendrycks et al.”
On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (2020, background): “Multiple recent works (Devlin et al., 2019; Phang et al., 2018; Lee et al., 2020; Zhu et al., 2020; Dodge et al., 2020; Pruksachatkun et al., 2020) that have observed BERT ﬁne-tuning to be unstable explicitly attribute this ﬁnding to the small number of training examples.”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “…test set in order to reveal test set results for any other target task, a large majority of papers that report test set numbers omit this metric and decline to report potentially unﬂattering bias numbers (Raffel et al., 2020; Pruksachatkun et al., 2020; Schick and Schütze, 2020; He et al., 2020).”
Exploring and Predicting Transferability across NLP Tasks (2020, background): “Concurrent work indicates that intermediate tasks that require high-level inference and reasoning abilities tend to work best (Pruksachatkun et al., 2020).”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, methodology): “We consider three ﬁne-tuning formats: (1) ﬁne-tuning on QuALITY data, (2) ﬁne-tuning on RACE and zero-shot evaluating on QuALITY, and (3) applying intermediate training (Phang et al., 2018; Pruksachatkun et al., 2020) by ﬁrst ﬁne-tuning on RACE and then ﬁne-tuning on QuALITY.”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “Prior work shows effective transfer from data-rich source tasks (Phang et al., 2019), those that require complex reasoning and inference (Pruksachatkun et al., 2020), or those that are similar to the target task (Vu et al., 2020).”
Dataless Knowledge Fusion by Merging Weights of Language Models (2022, unknown): “Multi-task learning has shown that leveraging information across domains or tasks can be beneficial if the data sets, data set size and algorithms are well selected (Phang et al., 2018; Pruksachatkun et al., 2020; Poth et al., 2021; Weller et al., 2022).”
On Transferability of Prompt Tuning for Natural Language Understanding (2021, background): “In the PLM era, some works propose to tune the PLMs on intermediate tasks (Phang et al., 2019; Pruksachatkun et al., 2020; Gururangan et al., 2020; Wang et al., 2019a; Vu et al., 2020; Poth et al., 2021) before fine-tuning on the specific target task and achieve certain benefits.”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “, 2019), tasks that require complex reasoning and inference (Pruksachatkun et al., 2020), and beneficial relationships”
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020, methodology): “These datasets are useful in three settings: evaluation (Williams et al., 2018; Rajpurkar et al., 2018; Zellers et al., 2019); pretraining (Phang et al., 2018; Conneau et al., 2018; Pruksachatkun et al., 2020); and as training data for downstream tasks (Trivedi et al., 2019; Portelli et al., 2020).”