UL2: Unifying Language Learning Paradigms

By Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier García, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng and 3 more. 2022

395 citations recorded by the source corpus.

Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized&unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5&GPT-like models across multiple diverse setups. By scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised finetuning based NLP tasks. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B also works well with chain-of-thought prompting and reasoning, making it an appealing choice for research into reasoning at a small to medium scale of 20B parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model, achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release Flax-based T5X checkpoints for the UL2 20B&Flan-UL2 20B.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “The fourth is UL2 20B (Tay et al., 2022), and the ﬁfth is Codex (Chen et al., 2021, code-davinci-002 in the OpenAI API).”
PaLM 2 Technical Report (2023, methodology): “Given the strong results of UL2 (Tay et al., 2023), we use a tuned mixture of different pre-training objectives in this model to train the model to understand different aspects of language.”
Teaching Small Language Models to Reason (2022, neutral): “Here, LLMs are models with at least tens of billions of parameters, such as PaLM 540B (Chowdhery et al., 2022), GPT-3 175B (Brown et al., 2020), or UL2 20B (Tay et al., 2022).”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “To leverage these capabilities to real applications, pre-trained LLM is further fine-tuned on specific tasks (Rad-ford et al., 2019; Chung et al., 2022; Tay et al., 2023).”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, methodology): “LLMs: We use the following LLMs in our experiments: PaLM 2 text-bison-001 [12], Flan-UL2 [45], Vicuna-33b-v1.”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “Overall model The encoder-decoder backbone is initialized from a variant of the UL2 [7] encoderdecoder model that uses 32B parameters.”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “Intrinsic prompt tuning finds a low-dimensional subspace shared by tuning prompts across diverse tasks.”
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023, background): “Besides, we report the best result in the corresponding paper for models that are not publicly available ( e.g., PaLM) and report the result based on the best prompt found for public models, which is consistent with the previous work (Wei et al., 2022; Kojima et al., 2022; Tay et al., 2022).”
Sentiment Analysis in the Era of Large Language Models: A Reality Check (2023, methodology): “• Larger models do not always guarantee superior performance, for instance, Flan-UL2 often performs comparably to the GPT-3.”
Learning Interactive Real-World Simulators (2023, methodology): “…on Language-Table uses a Vision Transformer architecture G/14 [5] to process images, and the encoder-decoder architecture of UL2 language model [76] for encoding task descriptions and decoding tokens which can represent language, control actions, or other values of interest (described below).”
BloombergGPT: A Large Language Model for Finance (2023, background): “Work also explored other important aspects of achieving a high-performing LLM, such as different training objectives (Tay et al., 2022b), multilingual models (Scao et al., 2022), more efficient and smaller models (Black et al., 2022), and finding data and parameter-efficient training sizes…”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “In contrast to training exclusively for text-to-video tasks, the generative model of LLMs in the language domain emphasizes a large pretraining stage to learn a foundation [6, 60] by examining pre-training tasks that extend beyond text-to-video generation.”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, background): “The impressive performance of GPT-3 and other LLMs (Scao et al., 2022; Tay et al., 2022; Thoppilan et al., 2022; Fedus et al., 2021; Hoffmann et al., 2022; Zeng et al., 2022) in few-shot learning scenarios is a major finding as this helps LLMs to be more efficient, making it possible to use…”
TIES-Merging: Resolving Interference When Merging Models (2023, background): “…model for each task has various drawbacks: (1) for each new application, a separate model has to be stored and deployed [17, 89], and (2) models trained in isolation cannot leverage information from related tasks to improve in-domain performance or out-of-domain generalization [66, 58, 75].”
Better & Faster Large Language Models via Multi-token Prediction (2024, background): “For Tay et al. (2022), it is technically possible to have a larger proportion but in practice, the settings used have between 15% and 25% of masked tokens.”