Do Transformers Need Deep Long-Range Memory?

By Jack W. Rae, Ali Razavi. 2020

45 citations recorded by the source corpus.

Deep attention models have advanced the modelling of sequential data across many domains. For language modelling in particular, the Transformer-XL — a Transformer augmented with a long-range memory of past activations — has been shown to be state-of-the-art across a variety of well-studied benchmarks. The Transformer-XL incorporates a long-range memory at every layer of the network, which renders its state to be thousands of times larger than RNN predecessors. However it is unclear whether this is necessary. We perform a set of interventions to show that comparable performance can be obtained with 6X fewer long range memories and better performance can be obtained by limiting the range of attention in lower layers of the network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Christopher R'e
5. Luke S. Zettlemoyer
6. Mohit Bansal
7. Quoc V. Le
8. Jianfeng Gao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yi Tay
3. Mostafa Dehghani
4. Christopher R'e
5. Andrew Zisserman
6. Dacheng Tao
7. Xiaogang Wang
8. Cordelia Schmid

In Prompting & Reasoning

1. Benjamin Van Durme
2. Yi Tay
3. Donald Metzler
4. Sebastian Ruder
5. Guanghui Qin

In Model Architectures

1. Tri Dao
2. Christopher R'e
3. Yi Tay
4. Mostafa Dehghani
5. Donald Metzler
6. Dara Bahri
7. Atri Rudra
8. Stefano Ermon

Citations in the corpus

• Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “This is motivated by our empirical finding that long range attention is only needed in the last few layers - see also (Rae and Razavi, 2020).”
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “For approximate attention, we compare against reference implementations of Reformer [49], Local Attention [65], Linformer Attention [81], Smyrf [18], and LongShortFormer (LSFormer) [91].”
• Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “However, it has been debated how much long-range signal such tasks actually encode (Rae & Razavi, 2020).”
• The NLP Task Effectiveness of Long-Range Transformers (2022, background): “Rae and Razavi (2020) find that Transformer-XL does not necessarily need long and deep contexts.”