What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

By Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, Colin Raffel. 2022

236 citations recorded by the source corpus.

Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pretraining objectives used across state-of-the-art models differ significantly, and there has been limited systematic comparison of these factors. In this work, we present a large-scale evaluation of modeling choices and their impact on zero-shot generalization. In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder-only and encoder-decoder), trained with two different pretraining objectives (autoregressive and masked language modeling), and evaluated with and without multitask prompted finetuning. We train models with over 5 billion parameters for more than 170 billion tokens, thereby increasing the likelihood that our conclusions will transfer to even larger scales. Our experiments show that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zero-shot generalization after purely unsupervised pretraining. However, models with non-causal visibility on their input trained with a masked language modeling objective followed by multitask finetuning perform the best among our experiments. We therefore consider the adaptation of pretrained models across architectures and objectives. We find that pretrained non-causal decoder models can be adapted into performant generative causal decoder models, using autoregressive language modeling as a downstream task. Furthermore, we find that pretrained causal decoder models can be efficiently adapted into non-causal decoder models, ultimately achieving competitive performance after multitask finetuning. Code and checkpoints are available at https://github.com/bigscience-workshop/architecture-objective.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Scaling Up Models and Data with t5x and seqio (2022, background): “External adopters include academic and commercial users of Cloud TPUs, such as portions of the the Big Science project (Wang et al., 2022).”
Emergent Abilities of Large Language Models (2022, neutral): “…that instruction-based ﬁnetuning only worked for 68B parameter or larger decoder-only models, Sanh et al. (2022) induced similar behavior in a 11B model with an encoder-decoder architecture, which typically has higher performance after ﬁnetuning than decoder-only architectures (Wang et al., 2022a).”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Furthermore, zero-shot generalization is the “natural” setting for autoregressive decoder-only models, in which they perform best (Wang et al., 2022).”
Scaling Instruction-Finetuned Language Models (2022, methodology): “Finally, whereas much of prior work ﬁnetunes on either zero-shot without exemplars (Zhong et al., 2021; Wei et al., 2021; Sanh et al., 2021; Wang et al., 2022a) or few-shot with exemplars (Ye et al., 2021; Wei et al., 2021; Min et al., 2022; Wang et al., 2022c), our process ﬁnetunes on a mixture of…”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[120] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel.”
Efficient Training of Language Models to Fill in the Middle (2022, background): “…causal decoder-based models at the largest scale is due to their superiority in open-ended text generation, in-context learning (using few-shot priming), pretraining computational efficiency [Wang et al., 2022], and to some extent historical precedence in successful scale-ups [Brown et al., 2020].”
What Language Model to Train if You Have One Million GPU Hours? (2022, methodology): “Our decision is motivated by the ﬁndings of Wang et al. (2022), which showed that decoder-only models combined with an autoregressive language modeling objective provide the best zero-shot generalization abilities immediately after pre-training.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “The other direction is also possible; a model trained with a masked language objective can be adapted with the causal language modeling objec-tive [Wang et al., 2022a, Lester et al., 2021].”
Language Models are General-Purpose Interfaces (2022, background): “Moreover, the underlying foundation models are conducive to ﬁnetuning because of bidirectional modeling (Wang et al., 2022b).”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “Task Adaptation With the advances in capabilities of pre-trained LMs, a large set of techniques for transferring or adapting these models to end tasks of interest have been developed (Wang et al., 2022).”
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages (2023, background): “Wang et al. (2022) conduct an extensive ablation study over architectures and objectives with the conclusion that decoder-only models with causal language modeling exhibit the strongest zero-shot generalization.”
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model (2022, background): “This suggests that mixed pre-training, and not necessarily additional multitask training as suggested by Wang et al. (2022), is the key to train strong seq2seq-based Large-scale Language Models (LLM).”
Large Language Models Struggle to Learn Long-Tail Knowledge (2022, methodology): “We ﬁrst train a baseline 4.8 billion parameter LM on C4, following the setup from Wang et al. (2022).”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, unknown): “In addition, we chose to examine the broad family of decoder-based, auto-regressive LMs, which have been shown to be extremely effective for many NLP tasks, including few-and zero-shot tasks (Wang et al., 2022).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “While we will focus on these canonical combinations of architecture and pre-training objective as seen in popular models, recent work attempts to disentangle the effects of the architecture from the objective and explores other combinations such as training a decoder-only model on MLMs [4, 189].”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “Transformers (Vaswani et al., 2017) have enabled task unification across many AI domains (Brown et al., 2020; Chen et al., 2022a;b; Lu et al., 2022; Wang et al., 2022c).”
A Cookbook of Self-Supervised Learning (2023, methodology): “MLM, along with span-infilling techniques, remains popular as a SSL objective for large language models [Raffel et al., 2020, Wang et al., 2022a, Tay et al., 2022].”