Big Little Transformer Decoder

By Sehoon Kim, Karttikeya Mangalam, Jitendra Malik, Michael W. Mahoney, A. Gholami, Kurt Keutzer. 2023

24 citations recorded by the source corpus.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. J. Ainslie
2. James Lee-Thorp
3. Sumit K. Sanghai
4. Michiel de Jong
5. Yury Zemlyanskiy

In Efficiency & Compression

1. James Lee-Thorp
2. Sumit K. Sanghai
3. J. Ainslie
4. Michiel de Jong
5. Yury Zemlyanskiy

Citations in the corpus

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, background): “Speculative sampling (Chen et al., 2023; Leviathan et al., 2022; Kim et al., 2023) ameliorates the memory bandwidth bottleneck by proposing multiple tokens with a smaller model which are then scored in parallel by a larger model.”