Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

By Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, Donald Metzler. 2022

130 citations recorded by the source corpus.

There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour? How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.

Subfields: Artificial Intelligence, Condensed Matter Physics, Statistical and Nonlinear Physics, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Scaling & Learning Theory

Citations in the corpus

Galactica: A Large Language Model for Science (2022, background): “The focus was on upstream perplexity, and work by Tay et al. (2022a) showed that this does not always correlate with downstream performance.”
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “, 2021), the relative value of advances in model architectures (Narang et al., 2021; Tay et al., 2022), the use of language models for generation tasks (Bender et al.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “Scaling has inspired new research across multiple fronts, e.g., scaling laws [Kaplan et al., 2020, Hoffmann et al., 2022, Tay et al., 2022a], emergent abilities [Wei † Work completed while at Google. et al., 2022a, Ganguli et al., 2022], reasoning capabilities [Wei et al., 2022b, Lewkowycz et al.,…”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Crucial to this success has been attention-based memory architectures like Transformers (Vaswani et al., 2017), which show power-law scaling in performance with the number of parameters (Tay et al., 2022).”
Reproducible Scaling Laws for Contrastive Language-Image Learning (2022, background): “Interestingly, upstream performance does not always correlate with downstream performance [72, 73].”
BloombergGPT: A Large Language Model for Finance (2023, background): “Work also explored other important aspects of achieving a high-performing LLM, such as different training objectives (Tay et al., 2022b), multilingual models (Scao et al., 2022), more efficient and smaller models (Black et al., 2022), and finding data and parameter-efficient training sizes…”
Ring Attention with Blockwise Transformers for Near-Infinite Context (2023, background): “• Optimal compute performance : While Ring Attention scales context length with device count while maintaining performance, optimizing low-level operations is required for achieving optimal compute performance.”
Scaling Data-Constrained Language Models (2023, background): “[101] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler.”
Training Trajectories of Language Models Across Scales (2022, background): “, 2022), architectures (Tay et al., 2022), biases (Tal et al.”