Long Range Arena: A Benchmark for Efficient Transformers

By Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler. 2020

917 citations recorded by the source corpus.

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. LRA paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at this https URL.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Mechanical Engineering, Model Architectures, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Mechanical Engineering

In Model Architectures

In Evaluation & Benchmarks

Citations in the corpus

Rethinking Attention with Performers (2020, methodology): “Performers obtain the largest LRA (Long Range Arena) score among all tested scalable Transformers methods (which we deﬁne by having speed of > 100 examples/sec).”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “FlashAttention enables the first Transformer that can achieve better-than-chance performance on the Path-X [77] challenge, solely from using a longer sequence length (16K).”
Self-attention Does Not Need $O(n^2)$ Memory (2021, methodology): “…and architectures with more favorable complexity classes (Kitaev et al., 2020; Roy et al., 2021; Zaheer et al., 2020; Choromanski et al., 2020; Wang et al., 2020; Ren et al., 2021; Child et al., 2019; Tay et al., 2021; Wang et al., 2020; Ma et al., 2021; Shen et al., 2021; Qiu et al., 2020).”
Perceiver: General Perception with Iterative Attention (2021, methodology): “…simplifying the matrix multiplications used in QKV attention (Choromanski et al., 2021; Peng et al., 2021; Ki-taev et al., 2020; Xiong et al., 2021; Katharopoulos et al., 2020; Tay et al., 2021a), or by introducing bottlenecks into each module’s computation (Lee et al., 2019; Wang et al., 2020b).”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “Compared to Performer [14], which is one of the fastest Transformer architectures (see [55]), the proposed shifted window based self-attention computation and the overall Swin Transformer architectures are slightly faster (see Table 5), while achieving +2.”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “(Computationally-efficient Transformer architectures (Tay et al., 2021) could potentially cut runtimes down substantially.”
Pretrained Transformers as Universal Computation Engines (2021, methodology): “Taken from (Nangia and Bowman 2018; Tay et al. 2020), the model is shown a sequence of list operations (ex.”
Efficient Transformers: A Survey (2020, methodology): “There have been recent attempts to unify evaluation on Eﬃcient Transformers, namely Long Range Arena, i.e., LRA, (Tay et al., 2021a) that benchmarked 10 diﬀerent xformer variants on long range modeling tasks.”
What Language Model to Train if You Have One Million GPU Hours? (2022, background): “Alternative attention mechanisms (Tay et al., 2020) or optimizers are examples of other dimensions that could be investigated, potentially leading to improved models.”
Nyströmformer: A Nystöm-based Algorithm for Approximating Self-Attention (2021, methodology): “The official LRA benchmark (Tay et al. 2020) is implemented in Jax/Flax (Frostig, Johnson, and Leary 2018).”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, background): “Readers may refer to [39] and [38] for a comprehensive survey and benchmarks, respectively.”
Random Feature Attention (2021, methodology): “We follow Tay et al. (2021) and consider sequences with 500–2,000 symbols.”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “Our Linear model is similar to the MLP-Mixer (Tolstikhin et al., 2021) (for vision) and also the Random Syn-thesizer (Tay et al., 2020a), but simpliﬁes the latter model further by removing the multiple heads and softmax projections, resulting in just two matrix multiplications in the mixing…”