Limits to Depth Efficiencies of Self-Attention Supplementary Material

By Yoav Levine. 2021

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Daniel Jannai
2. Amnon Shashua
3. Noam Wies
4. Ofir Press
5. Stella Biderman
6. Teven Le Scao
7. Julien Launay
8. Yoav Levine

Citations in the corpus

• What Language Model to Train if You Have One Million GPU Hours? (2022, background): “Configuration (3) is shallower than recommended by Levine et al. (2020), but delivers 3% higher throughput compared to (2).”
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention (2021, methodology): “However, it is important to note the vocabulary independent depth-to-width Transformer expressivity considerations given in (Levine et al., 2020): for a given parameter budget, a network can be too deep.”