Exploring and Predicting Transferability across NLP Tasks

By Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer. 2020

190 citations recorded by the source corpus.

Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. Can fine-tuning these models on tasks other than language modeling further improve performance? In this paper, we conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems (text classification, question answering, and sequence labeling). Our results show that transfer learning is more beneficial than previously thought, especially when target task data is scarce, and can improve performance even when the source task is small or differs substantially from the target task (e.g., part-of-speech tagging transfers well to the DROP QA dataset). We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task, and we validate their effectiveness in experiments controlled for source and target data size. Overall, our experiments reveal that factors such as source data size, task and domain similarity, and task complexity all play a role in determining transferability.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “MTL has been applied to many NLP scenarios in recent years primarily focusing on improving the performance on training tasks or to new domains by leveraging the signal from related tasks (Collobert and Weston, 2008; Raffel et al., 2020; Vu et al., 2020).”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “While prior work has shown the benefits of intermediate finetuning (Pruksachatkun et al., 2020; Vu et al., 2020) or multi-task finetuning (Aghajanyan et al.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “In many cases, it is not unreasonable to expect negative transfer (Rosenstein et al., 2005; Vu et al., 2020).”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “Understanding when datasets transfer well is still an open and active area of research (Vu et al. 2020; Pruksachatkun et al. 2020).”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “Efforts on the data side have focused on intermediate ﬁne-tuning either by using unlabeled target domain data (Karouzos et al., 2021; Gururangan et al., 2020) or via labeled data from other tasks (Phang et al., 2018; Agha-janyan et al., 2021; Vu et al., 2020).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, methodology): “…to learn from during upstream learning (Za- mir et al., 2018; Standley et al., 2020), potentially with task2vec methods (Achille et al., 2019; Vu et al., 2020); (3) applying task augmentation strate- gies to prevent over-fitting (Murty et al., 2021); (4) learning to accumulate knowledge…”
Memory Efficient Continual Learning with Transformers (2022, background): “There are a number of works that explores task transferability in NLP [42, 32, 59, 45, 44]”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, background): “) , which is a multi-layer perceptron consisting of two feed-forward layers and a ReLU non-linearity: where z τ ∈ R t can be a learnable parameter or any pretrained task features (Vu et al., 2020), and the task projector network h I ( .”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, methodology): “There have also been efforts to predict task transferability (Bingel and Søgaard, 2017; Vu et al., 2020; Poth et al., 2021).”
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022, methodology): “Prior work (Achille et al., 2019; Vu et al., 2020) formalizes the notion of task similarity using learned task embeddings so an another in terms of computational cost when it involves multi-task training on fewer examples (Appendix B.3) alternate formulation would be to divide the tasks into groups…”
Editing Models with Task Arithmetic (2022, unknown): “[91; 92], inter alia, explored strategies for representing tasks with continuous embeddings, in order to to predict task similarities and transferability, or to create taxonomic relations.”
Structured Prediction as Translation between Augmented Natural Languages (2021, methodology): “However, there have been recent advances in the attention mechanism that reduce the complexity to O ( L log L ) as in Reformer (Kitaev et al., 2020), or to O ( L ) as in Linformer (Wang et al., 2020).”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, background): “To address these issues, future work can take inspiration from work related to negative task interference (Wang et al., 2020a; Larson and Leach, 2022), transferability (Vu et al., 2020; Wu et al., 2020b; Xing et al., 2022) and lifelong learning (Wang et al., 2020b).”
On Transferability of Prompt Tuning for Natural Language Understanding (2021, background): “In the PLM era, some works propose to tune the PLMs on intermediate tasks (Phang et al., 2019; Pruksachatkun et al., 2020; Gururangan et al., 2020; Wang et al., 2019a; Vu et al., 2020; Poth et al., 2021) before fine-tuning on the specific target task and achieve certain benefits.”
Unsupervised Cross-Task Generalization via Retrieval Augmentation (2022, background): “Although there are some existing studies on task-to-task relatedness and transferability (Vu et al., 2020; Lange et al., 2021; Padmakumar et al., 2022), most of them are not designed for unsupervised settings and few are done with multi-task (prompted) upstream models.”
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020, background): “This setup generally involves ﬁne-tuning on a single task (Pruksachatkun et al., 2020; Vu et al., 2020) or multiple tasks (Liu et al., 2019a; Wang et al., 2019b; Raffel et al., 2019b), occasionally referred to as the intermediate task(s) , before ﬁne-tuning on the target task .”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, methodology): “tasks benefit the most from sequential fine-tuning while at the same time revealing the largest performance variances (Phang et al., 2018; Vu et al., 2020).”
Memory Efficient Continual Learning for Neural Text Classification (2022, background): “There are a number of works that explores task transferability in NLP (Phang et al., 2018; Liu et al., 2019; Vu et al., 2020; Puigcerver et al., 2020; Pruksachatkun et al., 2020).”
Merging Models with Fisher-Weighted Averaging (2021, background): “Recently, it has been shown that training on an “intermediate” task between pre-training and ﬁne-tuning can further boost performance through additional transfer of capabilities from the intermediate task [47, 60, 51, 48].”