What to Pre-Train on? Efficient Intermediate Task Selection

By Clifton A. Poth, Jonas Pfeiffer, Andreas Ruckl'e, Iryna Gurevych. 2021

108 citations recorded by the source corpus.

Intermediate task fine-tuning has been shown to culminate in large transfer gains across many NLP tasks. With an abundance of candidate datasets as well as pre-trained language models, it has become infeasible to experiment with all combinations to find the best transfer setting. In this work, we provide a comprehensive comparison of different methods for efficiently identifying beneficial tasks for intermediate transfer learning. We focus on parameter and computationally efficient adapter settings, highlight different data-availability scenarios, and provide expense estimates for each method. We experiment with a diverse set of 42 intermediate and 11 target English classification, multiple choice, question answering, and sequence tagging tasks. Our results demonstrate that efficient embedding based methods, which rely solely on the respective datasets, outperform computational expensive few-shot fine-tuning approaches. Our best methods achieve an average Regret@3 of 1% across all target tasks, demonstrating that we are able to efficiently identify the best datasets for intermediate training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “…transfer (Pfeiffer et al., 2020b, 2021b; Üstün et al., 2020; Vidoni et al., 2020; Ansell et al., 2021), community QA (Rücklé et al., 2020), and task composition for transfer learning (Stickland and Murray, 2019; Pfeiffer et al., 2021a; Lauscher et al., 2020; Wang et al., 2021; Poth et al., 2021).”
Memory Efficient Continual Learning with Transformers (2022, background): “[45] present a large-scale study on Adapter-based sequential fine-tuning.”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, methodology): “There have also been efforts to predict task transferability (Bingel and Søgaard, 2017; Vu et al., 2020; Poth et al., 2021).”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “human-annotated datasets for generic tasks (Phang et al., 2018; Pruksachatkun et al., 2020; Chang and Lu, 2021; Aghajanyan et al., 2021a; Poth et al., 2021) and text classification (Zhang and Zhang, 2021).”
Memory Efficient Continual Learning for Neural Text Classification (2022, background): “Poth et al. (2021) present a large-scale study on adapter-based sequential ﬁne-tuning.”
On Transferability of Prompt Tuning for Natural Language Understanding (2021, background): “In the PLM era, some works propose to tune the PLMs on intermediate tasks (Phang et al., 2019; Pruksachatkun et al., 2020; Gururangan et al., 2020; Wang et al., 2019a; Vu et al., 2020; Poth et al., 2021) before fine-tuning on the specific target task and achieve certain benefits.”
Hierarchical Inductive Transfer for Continual Dialogue Learning (2022, background): “Unlike conventional adapters 294 (Houlsby et al., 2019; Poth et al., 2021; Pfeiffer 295 et al., 2021), knowledge in the proposed adapters 296 will be used to further boost the learning of new 297 dialogue tasks, whereas knowledge of each task 298 is separated into general and task-specific parts…”
TIES-Merging: Resolving Interference When Merging Models (2023, background): “Moreover, it can be complex and resource-intensive to determine how best to mix datasets to ensure that multitask training is beneficial for all tasks [55, 54, 80, 52, 2, 17].”
Dataless Knowledge Fusion by Merging Weights of Language Models (2022, unknown): “Multi-task learning has shown that leveraging information across domains or tasks can be beneficial if the data sets, data set size and algorithms are well selected (Phang et al., 2018; Pruksachatkun et al., 2020; Poth et al., 2021; Weller et al., 2022).”