ETC: Encoding Long and Structured Data in Transformers

By J. Ainslie, Santiago Ontañón, Chris Alberti, Philip Pham, Anirudh Ravula, Sumit K. Sanghai. 2020

58 citations recorded by the source corpus.

Transformer-based models have pushed the state of the art in many natural language processing tasks. However, one of their main limitations is the quadratic computational and memory cost of the standard attention mechanism. In this paper, we present a new family of Transformer models, which we call the Extended Transformer Construction (ETC), that allows for significant increases in input sequence length by introducing a new global-local attention mechanism between a global memory and the standard input tokens. We also show that combining global-local attention with relative position encodings allows ETC to handle structured data with ease. Empirical results on the Natural Questions data set show the promise of the approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Big Bird: Transformers for Longer Sequences (2020, methodology): “Finally, our work is closely related to and built on the work of Extended Transformers Construction [4].”
Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, background): “Beltagy et al. (2020); Ainslie et al. (2020); Zaheer et al. (2020) allow a small number of special tokens having access to all key elements.”
Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “Thirdly, many papers tend to conﬂate the effectiveness of the inductive bias with the beneﬁts of pretraining (Ainslie et al., 2020; Preprint Zaheer et al., 2020; Wang et al., 2020), which tends to obfuscate the true value of the architecture.”
Efficient Transformers: A Survey (2020, methodology): “, 2020) O(n(k +m)) X FP+M ETC (Ainslie et al., 2020) O(N(2) g +NNg) 7 FP+M Synthesizer (Tay et al.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “where a single softmax is performed over the concatenated dimension of size (ĤŴ ,+N), inspired by ETC [1].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, background): “For example, Longformer[2], BigBird[47] and ETC[1] combine the sparsity and memory mechanisms; Synthesizers[36] combines the sparsity and low-rank mechanisms.”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, background): “, attending neighboring tokens within a constant window size, is another solution for both language [3, 78, 1] and vision [59, 44, 80].”
Recurrent Memory Transformer (2022, background): “Among them are Star-Transformer (Guo et al., 2019), Longformer (Beltagy et al., 2020), GMAT (Gupta and Berant, 2020), Extended Transformer Construction (ETC) (Ainslie et al., 2020) and Big Bird (Zaheer et al., 2020).”
Memory Transformer (2020, background): “Among the most recent transformers with global representations are Star-Transformer [11], Long-former [2], and Extended Transformer Construction (ETC) [1].”
k-means Mask Transformer (2022, background): “Transformer [90] and its variants [57,95,75,26,8,108,39,2] have advanced the state-of-the-art in natural language processing tasks [31,83,30] by capturing relations across modalities [4] or in a single context [25,90].”
RealFormer: Transformer Likes Residual Attention (2020, methodology): “…the genericity of Real-Former by using it as a drop-in replacement of two recent state-of-the-art Transformer variation models: ADMIN (Liu et al., 2020) from the Neural Machine Translation (NMT) domain, and ETC (Ainslie et al., 2020) that extends Transformer to handle long and structured inputs.”
LongT5: Efficient Text-To-Text Transformer for Long Sequences (2021, unknown): “Moreover, recent work in long-input transformers (Ainslie et al., 2020; Zaheer et al., 2020b; Beltagy et al., 2020; Tay et al., 2021) has shown that increasing the input length a Trans-former is able to process results in further performance gains.”
Focal Attention for Long-Range Interactions in Vision Transformers (2021, methodology): “It applies a standard Transformer, originally developed for NLP [57], to encode an image by analogously splitting the image into a sequence of visual tokens.”
Fine-tuning Image Transformers using Learnable Memory (2022, background): “In a similar vein, BigBird [45] and ETC [1] models introduce special global tokens that are not part of the input and serve as connector tokens for locally connected segments.”
Long Range Language Modeling via Gated State Spaces (2022, background): “Over the years, a variety of improvements have been proposed to alleviate this quadratic complexity (Tay et al., 2020; Choromanski et al., 2021; Ramsauer et al., 2021; Wang et al., 2020; Katharopoulos et al., 2020; Peng et al., 2021; Ainslie et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020; D”
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, background): “Transformer [80] variants [2, 7, 22, 30, 48, 66, 84, 92] have advanced the state-of-the-art in many natural language processing tasks [25, 26, 75] by capturing relations across modalities [3] or in a single context (self-attention) [21, 80].”