ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

By Vamsi Krishna Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni and 4 more. 2021

240 citations recorded by the source corpus.

Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “There has been a range of work in this domain (Yi et al., 2019; Mishra et al., 2021; Wei et al., 2021; Khashabi et al., 2020; Sanh et al., 2021; Aribandi et al., 2021), which differ in training and evaluation data, formatting of instructions, size of pretrained models, and other experimental…”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “In Section 2, we describe the multiple tasks and prompt repositories (Sanh et al., 2022; Wang et al., 2022; Wei et al., 2022a; Ye et al., 2021; Aribandi et al., 2022) that we unify to massively scale the number of tasks used for instruction-tuning.”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, methodology): “…wording of the instruction template, these prior LLMs mostly tune with template sets targeted to a single prompt setting: for zero-shot prompting (Wei et al., 2021; Sanh et al., 2021; Aghajanyan et al., 2021; Aribandi et al., 2021) or for few-shot prompting (Min et al., 2022; Wang et al., 2022c).”
Galactica: A Large Language Model for Science (2022, background): “The closest analog to this approach for large language models is ExT5 (Aribandi et al., 2021).”
UL2: Unifying Language Learning Paradigms (2022, methodology): “We leave the combination of massive multi-task training (Aribandi et al., 2021) and UL2 to future work.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “, 2020) and ExT5 (Aribandi et al., 2022) suggest that multi-task learning in pre-training can be more helpful than fine-tuning, we thus propose to include a variety of instruction prompted datasets including language understanding, generation, and information extraction in GLM-130B’s pre-training.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “, 2022) or better finetuning results on downstream tasks (McCann et al., 2018; Aribandi et al., 2022).”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “ExT5 (Aribandi et al., 2021) shows that scaling up multi-task learning helps improve pretraining efﬁciency and down-stream performances.”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, background): “ExT5 (Aribandi et al., 2021) applies multitask pretraining as well, but it focuses on multitask cotraining transfer instead of zero-shot generalization.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “…prior work, however, requires additional data such as aggregating dozens or hundreds of NLP datasets [Raffel et al., 2019, Aghajanyan et al., 2021, Aribandi et al., 2022], writing additional templates of instructions [Wei et al., 2021, Sanh et al., 2022], or ﬁnetuning on human-labeled annotations…”
Transformer Memory as a Differentiable Search Index (2022, neutral): “Finally, this work is also closely related to the theme of model unification where unified multi-task models have been recently popular due to immense potential (Raffel et al., 2019; Khashabi et al., 2020; Aribandi et al., 2021).”
A Survey of Large Language Models (2023, unknown): “1M ExMix [196] Nov-2021 11 107 18M UnifiedSKG [197] Jan-2022 6 21 812K Super Nat.”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, background): “There have been an number of works studying instruction following as a cross-task generalization across a pool of NLP tasks [41, 73, 54, 4, 72].”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “Compared to T5, ExT5 is continue pre-trained on C4 and ExMix, which is a combinition of 107 supervised NLP tasks across diverse domains.”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “We refer to this setting as “pre-finetuning” [1,66,6].”
Confident Adaptive Language Modeling (2022, background): “Recent advances in Large Language Models (LLMs) have led to breakthroughs in language understanding and language generation across almost every widely-used Natural Language Processing (NLP) task considered in the ﬁeld today [5; 15; 17; 20; 51; 52; 53; 75; 89; 73].”
Point Transformer V3: Simpler, Faster, Stronger (2023, background): “Deep learning models have experienced rapid advancements in various areas, such as 2D vision [24, 38, 78, 86] and natural language processing (NLP) [1, 37, 56, 79], with their progress often attributed to the effective utilization of scale, encompassing factors such as the size of datasets, the…”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “, 2022) or better finetuning results on downstream tasks (McCann et al., 2018; Liu et al., 2019; Aghajanyan et al., 2021; Aribandi et al., 2022).”
A Comprehensive Overview of Large Language Models (2023, unknown): “1.3 BIG-bench [282]: The BIG-bench (Behavior of Intelligent Generative Models Benchmark) is a large-scale benchmark designed to test the abilities of LLMs across a wide range of tasks, including reasoning, creativity, ethics, and understanding of specific domains.”