Muppet: Massive Multi-task Representations with Pre-Finetuning

By Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke S. Zettlemoyer, Sonal Gupta. 2021

301 citations recorded by the source corpus.

We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, background): “…a recent survey on MTL for NLP. Differing from prior MTL work which focuses on performance improvements across training tasks (Raffel et al., 2020; Aghajanyan et al., 2021) or to new domains (Axelrod et al., 2011), our work is motivated by improving zero-shot generalization to tasks that were…”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “…wording of the instruction template, these prior LLMs mostly tune with template sets targeted to a single prompt setting: for zero-shot prompting (Wei et al., 2021; Sanh et al., 2021; Aghajanyan et al., 2021; Aribandi et al., 2021) or for few-shot prompting (Min et al., 2022; Wang et al., 2022c).”
Scaling Instruction-Finetuned Language Models (2022, methodology): “• Aghajanyan et al. (2021) showed that a stage of massively-multitask ﬁnetuning (without instructions) improves performance on a range of downstream ﬁnetuning tasks for models such as RoBERTa (Liu et al., 2019) and BART (Lewis et al., 2019).”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, methodology): “This task collection also enables multi-task learning at a scale twice that of previous work (Aghajanyan et al., 2021).”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “Furthermore, in the context of multi-task learning, recent studies have shown tremendous promise in showing generalization to instances of similar tasks (Khashabi et al., 2020; Aghajanyan et al., 2021) (Table 2a; 1st column).”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “Cross-task generalization with PLMs Multitask learning and transfer learning go beyond task boundaries, view different tasks as related, and have been shown to outperform single-task learning (Aghajanyan et al., 2021a; Vu et al., 2021).”
MetaICL: Learning to Learn In Context (2021, background): “Prior work has shown that multi-task learning on a large collection of tasks leads to better performance on a new task, either when tested zero-shot (Khashabi et al., 2020; Zhong et al., 2021; Mishra et al., 2022; Wei et al., 2022) or when further ﬁnetuned (Aghajanyan et al., 2021; Ye et al., 2021).”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, methodology): “Relation to Meta/Multitask-Learning Our method is closely related to, but different from meta-learning (Yin, 2020; Murty et al., 2021) and multi-task learning (Ye et al., 2021; Aghajanyan et al., 2021).”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, background): “Numerous approaches have been proposed to ﬁll this gap, with an additional (intermediate) pre-training stage, mostly based on multi-task learning (Raffel et al., 2019; Aghajanyan et al., 2021).”
Transcending Scaling Laws with 0.1% Extra Compute (2022, positive): “A range of prior work has shown that ﬁnetuning language models on a collection of NLP tasks can improve downstream performance on a broad range of downstream tasks [Aghajanyan et al., 2021, Aribandi et al., 2022, Wei et al., 2021, Sanh et al., 2022, Ouyang et al., 2022, inter alia ].”
Meta-learning via Language Model In-context Tuning (2021, background): “Multi-task learning is shown to improve performance on various downstream tasks, especially tasks with small training sets (Khashabi et al., 2020; Ye et al., 2021; Aghajanyan et al., 2021).”
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative (2021, methodology): “, 2020; Aghajanyan et al., 2021). Specifically, our paper questions the wisdom of having disjoint further pre-training then fine-tuning steps on a generalist model. In response, we advocate for an alternative approach in which we directly introduce the end-task objective of interest into the learnin”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “We refer to this setting as “pre-finetuning” [1,66,6].”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “human-annotated datasets for generic tasks (Phang et al., 2018; Pruksachatkun et al., 2020; Chang and Lu, 2021; Aghajanyan et al., 2021a; Poth et al., 2021) and text classification (Zhang and Zhang, 2021).”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “, 2019), multi-task prefinetuning (Aghajanyan et al., 2021a), better design choices and training strategies (Mosbach et al.”