An Attention Free Transformer

By Shuangfei Zhai, Walter A. Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, J. Susskind. 2021

186 citations recorded by the source corpus.

We introduce Attention Free Transformer (AFT), an efficient variant of Transformers that eliminates the need for dot product self attention. In an AFT layer, the key and value are first combined with a set of learned position biases, the result of which is multiplied with the query in an element-wise fashion. This new operation has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible to both large input and model sizes. We also introduce AFT-local and AFT-conv, two model variants that take advantage of the idea of locality and spatial weight sharing while maintaining global connectivity. We conduct extensive experiments on two autoregressive modeling tasks (CIFAR10 and Enwik8) as well as an image recognition task (ImageNet-1K classification). We show that AFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “LambdaNetworks [2], AFT [97] and FLASH [44] are other attempts at replacing attention in the context of image classiﬁcation and language modeling.”
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, methodology): “Recent works [28, 38, 41, 57, 66] have proposed methods to lower the compute and memory requirements of vision transformers.”
RWKV: Reinventing RNNs for the Transformer Era (2023, methodology): “…LLaMA (Touvron et al., 2023), and Chinchilla (Hoffmann et al., 2022) Table 1: Complexity comparison with different Trans-formers: Reformer (Kitaev et al., 2020), Linear Trans-former (Katharopoulos et al., 2020), Performer (Choro-manski et al., 2020), AFT (Zhai et al., 2021), MEGA (Ma et al., 2023).”
Filter-enhanced MLP is All You Need for Sequential Recommendation (2022, background): “Its time complexity and receptive field are O(n2) and n [57, 64], respectively.”
ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis (2021, unknown): “A recent alternative to transformer models based on self-attention mechanisms is attention-free architectures based on feedforward models such as MLP-mixers or AFT [107, 108], suggested to offer on par performance to transformers on generic computer vision tasks.”
MambaOut: Do We Really Need Mamba for Vision?* (2024, unknown): “More recently, a new wave of RNN-like models has emerged [28, 29, 45, 65, 106], drawing significant interest from the community for their capability of paral-lelizable training and performing efficient inference on long sequences.”
Hyena Hierarchy: Towards Larger Convolutional Language Models (2023, methodology): “…time benchmarking different 2 layer models rather than changing the convolution parametrization: an order 2 Hyena, GSS (Mehta et al., 2022), H3 (Dao et al., 2022a), AFT-conv (Zhai et al., 2021), RWKV (Peng, 2021), and a standard GPT (Brown et al., 2020) using FlashAttention (Dao et al., 2022b).”