Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

By Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler. 2021

152 citations recorded by the source corpus.

There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Large Language Models

In Scaling & Learning Theory

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “Since it is typically only feasible to train these large models once, accurately estimating the best model hyperparameters for a given compute budget is critical (Tay et al., 2021).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “Recent work shows that good perplexity performance does not always align with good performance on downstream tasks (Tay et al., 2021).”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “We also hypothesize that modeling innovations that introduce inductive biases designed to exploit multi-task learning setups (Ha et al., 2016; Tay et al., 2021b) can push the boundary of the strong performance displayed by EXT5.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “This has become standard and defacto setup (Raffel et al., 2019; He et al., 2022; Tay et al., 2020; 2021b).”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Aside from the model size, [41] showed that the model shape also matters for efficient transfer from upstream pre-training to downstream finetuning.”
The Efficiency Misnomer (2021, methodology): “Token-Free Models Token-free models (Xue et al., 2021; Tay et al., 2021c; Jaegle et al., 2021) get rid of the large subword vocabulary by modeling at the character or byte level.”
What Language Model to Train if You Have One Million GPU Hours? (2022, background): “Moreover, as upstream performance is not always aligned with task performance (Tay et al., 2021), we must also measure downstream performance explicitly.”
Transformer Memory as a Differentiable Search Index (2022, background): “significant interest in recent years (Kaplan et al., 2020; Tay et al., 2021; Abnar et al., 2021).”
Designing Effective Sparse Expert Models (2022, methodology): “Furthermore, this work emphasizes jointly optimizing both the upstream pre-training and the downstream ﬁne-tuning metrics to avoid discrepancies (Tay et al., 2021).”
Training Trajectories of Language Models Across Scales (2022, background): “Previous work studies scaling on downstream tasks (Wei et al., 2022; Srivastava et al., 2022), pre-training data (Hernandez et al., 2022), architectures (Tay et al., 2022), biases (Tal et al., 2022), and other domains, such as vision tasks and neural machine translation (Alabdulmohsin et al., 2022).”
ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, methodology): “Furthermore, this work emphasizes jointly optimizing both the upstream pre-training and the downstream ﬁne-tuning metrics to avoid discrepancies (Tay et al., 2021).”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “Scaling Many existing studies investigate the scaling of neural networks to better understand scaling laws and optimize training budget (Hestness et al., 2017; Kaplan et al., 2020; Bornschein et al., 2020; Ghorbani et al., 2021; Bahri et al., 2021; Tay et al., 2021; Rae et al., 2021; Hoffmann et al.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “Transferring the scaling principles from large language models to the visual domain holds the potential for delivering consistent and expected performance improvements [92, 93, 43, 16, 94].”
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2022, background): “This finding is also generally supported by (Tay et al., 2021b) but discrepancies were found for the gap between pretraining and finetuning - highlighting the fact that observing downstream performance of large language model is indeed important.”
SCENIC: A JAX Library for Computer Vision Research and Beyond (2021, methodology): “On top of that, SCENIC has been used in numerous Google projects and research papers such as ViViT [6], OmniNet [28], TokenLearner [25], MBT [21], PolyViT [19], MTV [31], studies on scaling behaviour [5, 10, 29] and efficiency [9] of various models among others [15–18, 20].”
Scaling Data-Constrained Language Models (2023, unknown): “[94] Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler.”
BloombergGPT: A Large Language Model for Finance (2023, background): “Tay et al. (2021) reported that model shape (depth-width ratio) impacted performance on downstream tasks even if it had minimal impact on the pretraining objective.”
Unifying Language Learning Paradigms (2022, methodology): “Pre-training (or denoising) is generally applied on the subword level (Raffel et al., 2019; Devlin et al., 2018) but it is worth to note that it has also been applied on the character or byte-level (Xue et al., 2021; Tay et al., 2021c).”