Adaptively Sparse Transformers

By Gonçalo M. Correia, Vlad Niculae, André F. T. Martins. 2019

304 citations recorded by the source corpus.

Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are dense, assigning a non-zero weight to all context words. In this work, we introduce the adaptively sparse Transformer, wherein attention heads have flexible, context-dependent sparsity patterns. This sparsity is accomplished by replacing softmax with alpha-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the alpha parameter – which controls the shape and sparsity of alpha-entmax – allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, methodology): “Sparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020).”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “, the sparsemax [24] and its generalizations [27, 6, 3].”
Perceiver: General Perception with Iterative Attention (2021, methodology): “…et al., 2019), using non-local, non-dense attention patterns (Ho et al., 2019; Wang et al., 2020a; Belt-agy et al., 2020; Child et al., 2019; Correia et al., 2019; Ye et al., 2019; Roy et al., 2020), approximating or otherwise simplifying the matrix multiplications used in QKV attention…”
Reducing Transformer Depth on Demand with Structured Dropout (2019, background): “Others have proposed trainable adaptive mechanisms to control their memory footprint (Jernite et al., 2016; Sukhbaatar et al., 2019; Correia et al., 2019).”
BASE Layers: Simplifying Training of Large, Sparse Models (2021, background): “An orthogonal strand of work has improved the efficiency of transformer attention mechanisms, often by making them sparse (Child et al., 2019; Correia et al., 2019; Roy et al., 2020).”
Efficient Transformers: A Survey (2020, background): “…(Wang et al., 2020b) O ( N log N ) LP Luna (Ma et al., 2021) O ( kN ) M TokenLearner (Ryoo et al., 2021) O ( k 2 ) DS Adaptive Sparse Transformer (Correia et al., 2019) O ( N 2 ) Sparse Product Key Memory (Lample et al., 2019) O ( N 2 ) Sparse Switch Transformer (Fedus et al., 2021) O ( N 2…”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “It has been suggested that attention weights represent a model’s conﬁdence in detecting certain features [17, 85].”
ETC: Encoding Long and Structured Inputs in Transformers (2020, neutral): “Sparse attention has also been used to increase the inter-pretability of attention heads by allowing attention to assign exactly zero weight to certain input to-kens (Correia et al., 2019).”
Blockwise Self-Attention for Long Document Understanding (2019, background): “Most of them focus on the sparsiﬁcation of attention matrix, such as Star Transformer (Guo et al., 2019), Sparse Transformer (Child et al., 2019), Adaptive Sparse Transformer (Correia et al., 2019; Sukhbaatar et al., 2019), Log-Sparse Transformer (Li et al., 2019), etc.”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, background): “…such effect when the ﬁrst occurrence falls outside of the Transformer memory (ppl. of >2,700), highlighting the need to scale Transformers to even longer sequences (Child et al., 2019; Correia et al., 2019; Kitaev et al., 2020; Beltagy et al., 2020, inter alia ) to improve temporal generalization.”
Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models (2019, unknown): “al [6].”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, methodology): “These approaches include altering the attention mechanism (Correia et al., 2019), adding ℓ 1 penalties to neuron activations (Kasioumis et al., 2021; Georgiadis, 2019), pruning neurons (Frankle & Carbin, 2018), and using the softmax function as the non-linearity in the MLP layers (Elhage et al.,…”
A Practical Survey on Faster and Lighter Transformers (2021, methodology): “[20] introduced the Adaptively Sparse Transformer that replaces the Softmax by the α-entmax function, a diferentiable generalization of the Softmax that pushes small weights to be exactly zero.”
Transformers in Medical Imaging: A Survey (2022, methodology): “To discard the irrelevant spectral bands, they introduce a sparsity-based scheme [146].”
Rethinking Transformer-based Set Prediction for Object Detection (2020, methodology): “For future work, we would like to investigate the successful use of sparse attention mechanism [5, 6, 36] for directly modeling the relationship among multi-level features.”
Learning A Sparse Transformer Network for Effective Image Deraining (2023, background): “In principle, sparse attention can be categorized into data-based (fixed) sparse attention and content-based sparse attention [6, 38].”
Long Document Ranking with Query-Directed Sparse Transformer (2020, background): “The local attention serves as the backbone for many sparse transformer variations as it provides the basic local contextual information (Correia et al., 2019; Sukhbaatar et al., 2019; Beltagy et al., 2020).”