Adaptive Attention Span in Transformers

By Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, Armand Joulin. 2019

328 citations recorded by the source corpus.

We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, methodology): “Sparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020).”
Compressive Transformers for Long-Range Sequence Modelling (2019, background): “Several works have focused on reducing the computational cost of attention with sparse access mechanisms (Rae et al., 2016; Child et al., 2019; Sukhbaatar et al., 2019; Lample et al., 2019).”
Big Bird: Transformers for Longer Sequences (2020, background): “Prominently, Dai et al. [21], Sukhbaatar et al. [83], Rae et al. [74] have proposed auto-regresive models that work well for left-to-right language modeling but suffer in tasks which require bidirectional context.”
Reformer: The Efficient Transformer (2020, methodology): “In addition to standard methods such as precision reduction and gradient checkpointing (Sohoni et al., 2019), more efﬁcient versions of the Transformer model’s self-attention mechanism (Sukhbaatar et al., 2019a;b) have also recently been explored.”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “Other approaches include compressing along the sequence dimension to attend to multiple tokens at once [54, 59, 82, 93].”
Efficiently Scaling Transformer Inference (2022, neutral): “Finally, inference cost from the attention mechanism scales quadratically with input sequence length (Sukhbaatar et al., 2019; Choromanski et al., 2020; Dao et al., 2022).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “, 2019), adaptive spans (Sukhbaatar et al., 2019), and pruning (Fan, Grave, and Joulin, 2019) provide promising directions for creating compact but high performance models.”
Longformer: The Long-Document Transformer (2020, methodology): “This allows the top layers to learn higher-level representation of the entire sequence while having the (Dai et al., 2019) 41M - 1.06 Reformer (Kitaev et al., 2020) - - 1.05 Adaptive (Sukhbaatar et al., 2019) 39M 1.04 1.02 BP-Transformer (Ye et al., 2019) 38M - 1.02 Our Longformer 41M 1.02 1.00 We d”
Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “(5) Using the above grouping the attention weights can be calculated as: To build intuition, we note that using an empty L CMT , almost recovers the original formulation (with the exception of the existence of CMT token which can absorb part of the attention score).”
Perceiver: General Perception with Iterative Attention (2021, methodology): “…self-attention module, including using local or patchwise self-attention (Par-mar et al., 2018; Ramachandran et al., 2019; Zhao et al., 2020; Sukhbaatar et al., 2019), using non-local, non-dense attention patterns (Ho et al., 2019; Wang et al., 2020a; Belt-agy et al., 2020; Child et al.,…”
Fast Inference from Transformers via Speculative Decoding (2022, methodology): “These adaptive computation methods (e.g. Han et al., 2021; Sukhbaatar et al., 2019; Schuster et al., 2021; Scardapane et al., 2020; Bapna et al., 2020; Elbayad et al., 2019; Schwartz et al., 2020) aim to use less compute resources for easier inference steps.”
Reducing Transformer Depth on Demand with Structured Dropout (2019, background): “Others have proposed trainable adaptive mechanisms to control their memory footprint (Jernite et al., 2016; Sukhbaatar et al., 2019; Correia et al., 2019).”
Sparse is Enough in Scaling Transformers (2021, neutral): “Sparse attention-based approaches have made the attention layer more efﬁcient, especially for long sequences, by incorporating additional combinatorial mechanisms, as in [40], or selecting a subset of tokens this layer attends to [31, 5, 19, 37, 15, 4] or other approaches [12].”
Efficient Transformers: A Survey (2020, background): “This can be thought of as a form of parameter attention (Sukhbaatar et al., 2019b).”
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2020, background): “ dependencies beyond a ﬁxed length context without disrupting the temporal coherence. However, maintaining previous contexts in memory introduces signiﬁcant additional computational cost. In contrast,Sukhbaatar et al. (2019) extended the context length signiﬁcantly by learning the optimal attention ”
ETC: Encoding Long and Structured Inputs in Transformers (2020, background): “…2019; Lan et al., 2019; Raffel et al., 2019) have yielded state-of-the-art results in many NLP tasks such as language modeling (Child et al., 2019; Sukhbaatar et al., 2019; Rae et al., 2019; Kitaev et al., 2020), question answering (Lan et al., 2019; Beltagy et al., 2020), and summarization…”
Adaptively Sparse Transformers (2019, background): “At the heart of the Transformer While the sparse Transformer (Child et al., 2019) and the adaptive span Transformer (Sukhbaatar et al., 2019) only attend to words within a contiguous span of the past tokens, our model is not only able to obtain different and not necessarily contiguous sparsity…”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Other authors have worked on lightweight architectures, by modifying existing models (Zhang et al., 2018; Wu et al., 2019; Sukhbaatar et al., 2019a) or developing new networks, such as MobileNet (Howard et al., 2019), ShufﬂeNet (Zhang et al., 2017), and EfﬁcientNet (Tan & Le, 2019) in vision.”
Blockwise Self-Attention for Long Document Understanding (2019, background): “Most of them focus on the sparsiﬁcation of attention matrix, such as Star Transformer (Guo et al., 2019), Sparse Transformer (Child et al., 2019), Adaptive Sparse Transformer (Correia et al., 2019; Sukhbaatar et al., 2019), Log-Sparse Transformer (Li et al., 2019), etc.”