Synthesizer: Rethinking Self-Attention in Transformer Models

By Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, Che Zheng. 2020

76 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, methodology): “To address this problem, a wide variety of efficient alternatives and approximations have been proposed over the past couple of years (Tay et al., 2020b; Fournier et al., 2021).”
Perceiver: General Perception with Iterative Attention (2021, methodology): “…simplifying the matrix multiplications used in QKV attention (Choromanski et al., 2021; Peng et al., 2021; Ki-taev et al., 2020; Xiong et al., 2021; Katharopoulos et al., 2020; Tay et al., 2021a), or by introducing bottlenecks into each module’s computation (Lee et al., 2019; Wang et al., 2020b).”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “We explore the factorized, dense, and random Synthesizer variants from Tay et al. (2020), where self-attention is replaced with “synthetic attention” patterns.”
Efficient Transformers: A Survey (2020, methodology): “…et al., 2020b) O ( B 2 ) LP Longformer (Beltagy et al., 2020) O ( n ( k + m )) FP+M ETC (Ainslie et al., 2020) O ( N 2 g + NN g ) FP+M Synthesizer (Tay et al., 2020a) O ( N 2 ) LR+LP Performer (Choromanski et al., 2020a) O ( N ) KR Funnel Transformer (Dai et al., 2020) O ( N 2 ) FP+DS Linformer…”
Transformers in Vision: A Survey (2021, methodology): “Numerous methods have been proposed that make special design choices to perform self-atention more ‘eiciently’, for instance employing pooling/downsampling in self-atention [61, 209, 221], local window-based atention [138, 201], axial-atention [55, 83], low-rank projection atention [192, 208, 225], ”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “Our Linear model is similar to the MLP-Mixer (Tolstikhin et al., 2021) (for vision) and also the Random Syn-thesizer (Tay et al., 2020a), but simpliﬁes the latter model further by removing the multiple heads and softmax projections, resulting in just two matrix multiplications in the mixing…”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “The self-attention layer incurs considerable computational overhead at inference time; yet, its importance has often been questioned (Kovaleva et al., 2019; Tay et al., 2020; Raganato et al., 2020).”
Are Pretrained Convolutions Better than Pretrained Transformers? (2021, background): “Firstly, convolutions do not suffer from the quadratic memory complexity of self-attention - a problem signiﬁcant enough that it spawned the creation of the entirely new category of “efﬁcient” Transformer architectures (Tay et al., 2020b, 2021).”
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022, background): “Several recent works have explored mixing mechanisms, such as matrix multiplications (Tay et al., 2020; Lee-Thorp et al., 2021), MLP blocks (Tolstikhin et al., 2021; Liu et al., 2021), and spectral transforms (Lee-Thorp et al., 2021), as an efﬁcient replacement of attention in Transformer-like…”
A Survey of Transformers (2021, methodology): “Synthesizer [131] proposes to replace generated attention scores with: (1) a learnable, randomly initialized attention scores, and (2) attention scores output by a feed-forward network that is only conditioned on the querying input itself.”
An Attention Free Transformer (2021, methodology): “We also compare to several efficient Transformer baselines, namely Reformer [8], Synthesizer [12] , Linear Transformer [11] and Performer [13].”
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, unknown): “Tay et al. (2020) directly predicted both word and position contributions to the attention matrix without depending on token-to-token interactions.”
The Benchmark Lottery (2021, background): “An example of such a benchmark is the Long Range Arena benchmark [Tay et al., 2020b], whose usage continues to increase leading to the emergence of new ideas for improving the efﬁciency of Transformer models on long sequences.”