Effective Theory of Transformers at Initialization

By Emily Dinan, Sho Yaida, Susan Zhang. 2023

20 citations recorded by the source corpus.

We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Ali Farhadi
3. Ludwig Schmidt
4. Tim Dettmers
5. Mitchell Wortsman
6. Percy Liang
7. Mohit Bansal
8. Hannaneh Hajishirzi

In Statistical and Nonlinear Physics

1. Ali Farhadi
2. Ludwig Schmidt
3. Luke S. Zettlemoyer
4. Tim Dettmers
5. Ari S. Morcos
6. Mitchell Wortsman

In Model Architectures

1. Ali Farhadi
2. Ludwig Schmidt
3. Luke S. Zettlemoyer
4. Tim Dettmers
5. Ari S. Morcos
6. Mitchell Wortsman

In Scaling & Learning Theory

1. Ali Farhadi
2. Luke S. Zettlemoyer
3. Tim Dettmers
4. Ari S. Morcos
5. Mitchell Wortsman

Citations in the corpus

• Stable and low-precision training for large-scale vision-language models (2023, background): “Second, we do not examine the impact on stability of width scalings for initialization and training hyperparameters such as those examined by [19].”