Pay Attention when Required
By Swetha Mandava, Szymon Migacz, Alex Fit-Florea. 2020
11 citations recorded by the source corpus.
Transformer-based models consist of interleaved feed-forward blocks - that capture content meaning, and relatively more expensive self-attention blocks - that capture context meaning. In this paper, we explored trade-offs and ordering of the blocks to improve upon the current Transformer architecture and proposed PAR Transformer. It needs 35% lower compute time than Transformer-XL achieved by replacing ~63% of the self-attention blocks with feed-forward blocks, and retains the perplexity on WikiText-103 language modelling benchmark. We further validated our results on text8 and enwiki8 datasets, as well as on the BERT model.
Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Charlie Nash
- Peter W. Battaglia
- Jacob Menick
- Sander Dieleman
- Josh Tenenbaum
- Yoshua Bengio
- Sergey Levine
- Jessica B. Hamrick
In Model Architectures
In Efficiency & Compression
Citations in the corpus
- Generating Images with Sparse Representations (2021, methodology): “PAR Transformer Mandava et al. (2020) showed through Neural architecture search that the default alternation of fully connected and self attention layers in Transformer blocks is sub-optimal with respect to performance-speed trade offs.”