ETC: Encoding Long and Structured Inputs in Transformers

By J. Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Kenneth Fisher, Philip Pham, Anirudh Ravula, Sumit K. Sanghai, Qifan Wang, Li Yang. 2020

415 citations recorded by the source corpus.

Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Longformer: The Long-Document Transformer (2020, background): “lar, ETC (Ainslie et al., 2020) uses a similar local + global attention instead of full self-attention to scale Transformers to long documents.”
Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, background): “Beltagy et al. (2020); Ainslie et al. (2020); Zaheer et al. (2020) allow a small number of special tokens having access to all key elements.”
Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “Thirdly, many papers tend to conﬂate the effectiveness of the inductive bias with the beneﬁts of pretraining (Ainslie et al., 2020; Preprint Zaheer et al., 2020; Wang et al., 2020), which tends to obfuscate the true value of the architecture.”
Efficient Transformers: A Survey (2020, methodology): “…(Rae et al., 2020) O ( N 2 ) RC Sinkhorn Transformer (Tay et al., 2020b) O ( B 2 ) LP Longformer (Beltagy et al., 2020) O ( n ( k + m )) FP+M ETC (Ainslie et al., 2020) O ( N 2 g + NN g ) FP+M Synthesizer (Tay et al., 2020a) O ( N 2 ) LR+LP Performer (Choromanski et al., 2020a) O ( N ) KR…”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, background): “Later, the complexity of transformer attention has been reduced [49, 90], by introducing local [68] or sparse attention [23], together with a global memory [6, 103, 31, 1].”
Pre-Trained Models: Past, Present and Future (2021, background): “Another kind of methods (Guo et al., 2019; Lee et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020) combine global and local attention mechanisms, and then use global nodes to gather tokens in a sequence.”
Random Feature Attention (2021, background): “Another line of work aims to store past context into a memory module with limited size (Lee et al., 2019; Ainslie et al., 2020; Rae et al., 2020, inter alia ), so that accessing longer history only moderately increases the overhead.”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “For example, in models such as Longformer (Beltagy et al., 2020), ETC (Ainslie et al., 2020), and Big-Bird (Zaheer et al., 2020), attention is O ( N ) as a function of the input length, but quadratic in the number of “global tokens”; the latter must be sufﬁ-ciently large to ensure good performance.”
XCiT: Cross-Covariance Image Transformers (2021, background): “Other efﬁcient attention methods rely on local attention and adding a small number of global tokens, thus allowing interaction among all tokens only by hopping through the global tokens [1, 5, 34, 80].”
Luna: Linear Unified Nested Attention (2021, background): “, 2020), and low-rank or compressed context (Lee et al., 2019; Wang et al., 2019; Ainslie et al., 2020).”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “The empirical advantages of other pretraining tasks are more task-speciﬁc, for example, entity related masks for knowledge intensive applications [20, 24], and sequence-level tasks for long form text modeling [42].”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “HTML-T5 has an encoder-decoder architecture and is specialized to capture the structure – syntax and semantics – of long HTML pages better by adopting local and global attention mechanisms [3, 22] in the encoder.”
MATE: Multi-view Attention for Table Transformer Efficiency (2021, methodology): “Although some work addresses this limitation (Ainslie et al., 2020; Kitaev et al., 2020; Zaheer et al., 2020), there has been little prior work on scalable Transformer architectures for semi-structured text.1 However, although some of the more widely used benchmark tasks involving semi-structured…”
Generative Adversarial Transformers (2021, background): “Meanwhile, the NLP community has witnessed a major revolution with the advent of the Transformer network [65], a highly-adaptive architecture centered around relational attention and dynamic interaction.”
RealFormer: Transformer Likes Residual Attention (2020, methodology): “…the genericity of Real-Former by using it as a drop-in replacement of two recent state-of-the-art Transformer variation models: ADMIN (Liu et al., 2020) from the Neural Machine Translation (NMT) domain, and ETC (Ainslie et al., 2020) that extends Transformer to handle long and structured inputs.”
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, background): “…we note that this middle portion of the model could easily be replaced with any other sequence-to-sequence model including those with better compute performance such as Performer (Choromanski et al., 2021), Big Bird (Zaheer et al., 2020), RFA (Peng et al., 2021), ETC (Ainslie et al., 2020), etc.”