Scale Efficiently: Insights from Pretraining and Finetuning Transformers

By Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler. 2022

70 citations recorded by the source corpus.

Subfields: Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Gabriel Ilharco
2. Suchin Gururangan
3. Julien Launay
4. Sanjeev Arora
5. Siva Reddy
6. Ludwig Schmidt
7. Payel Das
8. Sadhika Malladi

In Scaling & Learning Theory

1. Sanjeev Arora
2. Mitchell Wortsman
3. Yuanzhi Li
4. Tengyu Ma
5. Jason D. Lee
6. Simon Shaolei Du
7. Colin Wei
8. Wei Hu

Citations in the corpus

• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Perplexity alone can be at odds with end-task performance (Tay et al., 2021), and modern works on LLMs predominantly report zero-shot performance (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022).”
• The Impact of Positional Encoding on Length Generalization in Transformers (2023, background): “But these studies use language modeling perplexity as the sole evaluation metric which does not shed light on downstream task performance (Tay et al., 2022).”
• LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, background): “Although perplexity aligns well with task performance in base LLMs (Xia et al., 2023a), but the relationship is much more obscure in fine-tuning tuning or instruction tuning (Tay et al., 2021; Zhou et al., 2023).”
• Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, unknown): “Tay et al. (2022) and (Liu et al., 2024) have shown that Deep & Narrow language models have better downstream performance than their shallower counterparts, at the expense of slower inference.”
• Reproducible Scaling Laws for Contrastive Language-Image Learning (2022, methodology): “Specifically, researchers have postulated scaling laws in the form of power law relationships between model performance and model compute, or data scale [35, 61, 73, 84].”