Transcending Scaling Laws with 0.1% Extra Compute

By Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q. Tran, David R. So, Siamak Shakeri, Xavier García, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery and 6 more. 2022

78 citations recorded by the source corpus.

Scaling language models improves performance but comes with significant computational costs. This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. The key idea is to continue training a state-of-the-art large language model (e.g., PaLM) on a few more steps with UL2's mixture-of-denoiser objective. We show that, with almost negligible extra computational costs and no new sources of data, we are able to substantially improve the scaling properties of large language models on downstream metrics. In this paper, we continue training PaLM with UL2R, introducing a new set of models at 8B, 62B, and 540B scale which we call U-PaLM. Impressively, at 540B scale, we show an approximately 2x computational savings rate where U-PaLM achieves the same performance as the final PaLM 540B model at around half its computational budget (i.e., saving $\sim$4.4 million TPUv4 hours). We further show that this improved scaling curve leads to 'emergent abilities' on challenging BIG-Bench tasks -- for instance, U-PaLM does much better than PaLM on some tasks or demonstrates better quality at much smaller scale (62B as opposed to 540B). Overall, we show that U-PaLM outperforms PaLM on many few-shot setups, i.e., English NLP tasks (e.g., commonsense reasoning, question answering), reasoning tasks with chain-of-thought (e.g., GSM8K), multilingual tasks (MGSM, TydiQA), MMLU and challenging BIG-Bench tasks. Finally, we provide qualitative examples showing the new capabilities of U-PaLM for single and multi-span infilling.

Subfields: Artificial Intelligence, Efficiency & Compression, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Scaling & Learning Theory

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Furthermore, they can be more efficiently adapted after pretraining to a non-causal architecture and objective–an approach which has been further explored and confirmed by Tay et al. (2022).”
Emergent Abilities of Large Language Models (2022, neutral): “Similar logic may apply for multi-step or arithmetic reasoning problems, where models are only scored on whether they get the ﬁnal answer to a multi-step problem correct, without any credit given to partially correct solutions.”
Scaling Instruction-Finetuned Language Models (2022, methodology): “Finally, we instruction-ﬁnetune U-PaLM , which is a 540B PaLM model initialized from PaLM-540B and then pretrained with an UL2 objective for 20k additional steps (Tay et al., 2022a,b).”
A Comprehensive Overview of Large Language Models (2023, unknown): “quently, researchers proposed several architectural designs and training strategies that showed superior performance compared to GPT-3 across various tasks [15], [14], [20], [21].”
Grounding Language Models to Images for Multimodal Generation (2023, background): “In future work, it will be interesting to test this, and trained models may learn additional interesting emergent behaviors (Wei et al., 2022).”
A Survey of Large Language Models (2023, unknown): “Instead of pre-training from scratch, a practical suggestion is to continually train casual decoders and then convert them into prefix decoders for accelerating convergence [29], e.g., U-PaLM [97] is derived from PaLM [56].”
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages (2023, methodology): “Based on the recent success with Preﬁx-LM (Tay et al., 2022b), we hypothesize that the trained Preﬁx-LM yields a competitive encoder representation that can be used for a range of discriminative tasks, while retaining generative capability as the decoder.”
Large Language Models: A Survey (2024, unknown): “The U-PaLM models of 8B, 62B, and 540B scales are continually trained on PaLM with UL2R, a method of continue training LLMs on a few steps with UL2’s mixture-of-denoiser objective [73].”