Synthesizer: Rethinking Self-Attention for Transformer Models

By Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, Che Zheng. 2020

407 citations recorded by the source corpus.

The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose \textsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexity by a relative $3.5\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “…(Lan et al., 2019; Tay et al., 2019; Zhang et al., 2021), efficient attention (Tay et al., 2020c; Kitaev et al., 2020; Choromanski et al., 2020; Tay et al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al., 2019), alternative non Transformer-based architectures…”
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, methodology): “To address this problem, a wide variety of efficient alternatives and approximations have been proposed over the past couple of years (Tay et al., 2020b; Fournier et al., 2021).”
Perceiver: General Perception with Iterative Attention (2021, methodology): “…simplifying the matrix multiplications used in QKV attention (Choromanski et al., 2021; Peng et al., 2021; Ki-taev et al., 2020; Xiong et al., 2021; Katharopoulos et al., 2020; Tay et al., 2021a), or by introducing bottlenecks into each module’s computation (Lee et al., 2019; Wang et al., 2020b).”
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (2021, neutral): “Some recent works have approximated the attention matrix with low-rank factorizations (Wang et al., 2020; Tay et al., 2020) or kernel methods (Katharopoulos et al.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “Figure 9 also shows that the Primer modifications are compatible with other efficient model families, such as large sparse mixture-of-experts like Switch Transformer [8] and efficient Transformer approximations like Synthesizer [23].”
The Efficiency Misnomer (2021, background): “That said, it is still common practice to parameter-match models to make ‘fair’ comparisons (Mehta et al., 2020; Lee-Thorp et al., 2021; Tay et al., 2020a; Xue et al., 2021; Wightman et al., 2021), even if one model is in reality slower or faster than another.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “We explore the factorized, dense, and random Synthesizer variants from Tay et al. (2020), where self-attention is replaced with “synthetic attention” patterns.”
Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “For the sake of brevity, we refer interested readers to (Tay et al., 2020c) for a detailed survey of this line of research.”
Transformers in Vision: A Survey (2021, methodology): “Numerous methods have been proposed that make special design choices to perform self-attention more ‘eiciently’, for instance employing pooling/downsampling in self-attention [61, 209, 221], local window-based attention [138, 201], axial-attention [55, 83], low-rank projection attention [192, 208, 2”
MLP-Mixer: An all-MLP Architecture for Vision (2021, background): “Ramachandran et al. [37], Tay et al. [47], Li et al. [26], and Bello [3] design networks with new attention-like mechanisms.”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, background): “For example, Longformer[2], BigBird[47] and ETC[1] combine the sparsity and memory mechanisms; Synthesizers[36] combines the sparsity and low-rank mechanisms.”
Random Feature Attention (2021, background): “3 , × Importantly, compared to the only two baselines that perform comparably to the baseline transformer model (Tay et al., 2020a; Zaheer et al., 2020), R FA has a clear advantage in both speed and memory efﬁciency, and is the only model that is competitive in both accuracy and efﬁciency. varying…”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “Tay et al. (2020d) survey many of the recent efficient attention works, which they broadly categorize as follows:”
cosFormer: Rethinking Softmax in Attention (2022, positive): “As shown in Table 5 and Figure 1, most pattern based methods (Beltagy et al., 2020; Zaheer et al., 2020; Tay et al., 2020a; 2021) and vanilla transformer (Vaswani et al., 2017) are much slower and require greater memory than COS F ORMER prevents them from extending to longer sequence.”
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections (2020, methodology): “Third, in lieu of recent work that downplays the effectiveness of QKV transformations [Kitaev et al., 2020, Tay et al., 2020], we do not attempt to apply HyperGrid to the self-attention projections.”
Are Pretrained Convolutions Better than Pretrained Transformers? (2021, background): “Firstly, convolutions do not suffer from the quadratic memory complexity of self-attention - a problem signiﬁcant enough that it spawned the creation of the entirely new category of “efﬁcient” Transformer architectures (Tay et al., 2020b, 2021).”
Pay Attention to MLPs (2021, background): “This baseline is a straightforward variant of Transformers without self-attention, which can also be viewed as a Random Synthesizer [40].”
Learning Graph Structures With Transformer for Multivariate Time-Series Anomaly Detection in IoT (2021, background): “To generalize the global-ﬁxed attention pattern proposed in [50], we apply a parameter matrix S ∈ R m × m ( m > n ) as a learnable global compatibility function across all training samples following the Synthesizer [51].”
CoCon: A Self-Supervised Approach for Controlled Text Generation (2020, background): “Transformer-based [29, 28] pretrained language models (LMs) have led a wave of new advances in natural language processing tasks as a means to extract contextualized word embeddings [6, 4, 32] and as text generators [24].”