Efficient Attention using a Fixed-Size Memory Representation

By Denny Britz, Melody Y. Guan, Minh-Thang Luong. 2017

32 citations recorded by the source corpus.

The standard content-based attention mechanism typically used in sequence-to-sequence models is computationally expensive as it requires the comparison of large encoder and decoder states at each time step. In this work, we propose an alternative attention mechanism based on a fixed size memory representation that is more efficient. Our technique predicts a compact set of K attention contexts during encoding and lets the decoder compute an efficient lookup that does not need to consult the memory. We show that our approach performs on-par with the standard attention mechanism while yielding inference speedups of 20% for real-world translation tasks and more for tasks with longer sequences. By visualizing attention scores we demonstrate that our models learn distinct, meaningful alignments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Irwan Bello
3. Yoshua Bengio
4. Ruslan Salakhutdinov
5. Luke S. Zettlemoyer
6. Mohit Bansal
7. Alec Radford
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Trevor Darrell
4. Dacheng Tao
5. Ming-Hsuan Yang
6. Jitendra Malik
7. Alan L. Yuille
8. Andrea Vedaldi

In Model Architectures

1. Irwan Bello
2. Alec Radford
3. Ilya Sutskever
4. Rewon Child
5. Franccois Fleuret
6. Apoorv Vyas
7. Angelos Katharopoulos
8. Scott Gray

In Efficiency & Compression

1. Irwan Bello
2. Ilya Sutskever
3. Alec Radford
4. Scott Gray
5. Franccois Fleuret
6. Apoorv Vyas
7. Angelos Katharopoulos
8. Rewon Child

Citations in the corpus

• Generating Long Sequences with Sparse Transformers (2019, methodology): “Outside of generative modeling, there are several works relevant to improving the efficiency of attention based off chunking (Chiu & Raffel, 2017) or using fixed length representations (Britz et al., 2017).”
• LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “Linear attention mechanisms date back to de Brébisson & Vincent (2016); Britz et al. (2017) and were later introduced in the visual domain by Chen et al. (2018); Shen et al. (2018).”
• Fast Transformers with Clustered Attention (2020, background): “[4] that propose to aggregate the information of the input sequence into fewer vectors and perform attention with these fewer vectors, thus speeding up the attention computation and reducing the memory requirements.”