The Depth-to-Width Interplay in Self-Attention.
By Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, Amnon Shashua. 2020
33 citations recorded by the source corpus.
Subfields: Model Architectures, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Model Architectures
- Jean-Baptiste Lespiau
- Sebastian Borgeaud
- Hoo-Chang Shin
- Geoffrey Irving
- Laurent Sifre
- Charlie Chen
- John M. Jumper
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “Levine et al. (2020) investigates the optimal depth-to-width ratio for a variety of standard model sizes.”
- Accelerating Large Language Model Decoding with Speculative Sampling (2023, background): “It has been observed that the performance of language models is relatively robust to changes in model aspect ratio (Levine et al., 2020), so this allows us to serve a powerful draft model which can be sampled rapidly on the same hardware as the target model.”
- A large language model for electronic health records (2022, methodology): “We developed GatorTron models with three configurations and determined the number of layers, hidden sizes, and number of attention heads according to the guidelines for optimal depth-to-width parameter allocation proposed by Levin et al 47 as well as our previous experience in developing BioMegatron”
- A study of generative large language model for medical research and healthcare (2023, unknown): “39 as well as our previous experience in developing GatorTron 15 .”