Compressive Transformers for Long-Range Sequence Modelling

By Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Timothy P. Lillicrap. 2019

900 citations recorded by the source corpus.

We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “• Language modelling on LAMBADA, Wikitext103 (Mer-ity et al., 2017), C4 (Raﬀel et al., 2020a), PG-19 (Rae et al., 2020) and the Pile (Gao et al., 2020).”
Rethinking Attention with Performers (2020, methodology): “Furthermore, we use PG-19 (Rae et al., 2020) as an alternative (B) pretraining benchmark, as it is made for long-length sequence training compared to the (now publicly unavailable) BookCorpus (Zhu et al.”
Big Bird: Transformers for Longer Sequences (2020, background): “Prominently, Dai et al. [21], Sukhbaatar et al. [83], Rae et al. [74] have proposed auto-regresive models that work well for left-to-right language modeling but suffer in tasks which require bidirectional context.”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “One can also attend over the states from previous sequences to help lengthen the context (e.g., Transformer-XL [15] and Compressive Transformer [72]).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, neutral): “It consists of data from 22 data sources, coarsely broken down into 5 categories: • Academic Writing: Pubmed Abstracts and PubMed Central, arXiv, FreeLaw,5 USPTO Backgrounds,6 PhilPapers,7 NIH Exporter8 • Web-scrapes and Internet Resources: CommonCrawl, OpenWebText2, StackExchange,9 Wikipedia (Engli”
Longformer: The Long-Document Transformer (2020, methodology): “For large models, given how expensive these experiments are, and following recent work (Ki-taev et al., 2020; Rae et al., 2020 et al., 2020) are not good ﬁt for the pretraining-ﬁnetuning paradigm as discussed in § 2.”
Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, background): “We first evaluate the efficacy of retrieving earlier blocks on two language modeling tasks which can be expected to have long-range token interactions: English language books (PG-19) [29] (3.7B tokens), and math papers from arXiv (5.6B tokens).”
Adaptive Semiparametric Language Models (2021, background): “, 2020), and (c) compressive transformer (Rae et al., 2020).”
Perceiver: General Perception with Iterative Attention (2021, methodology): “More broadly, cross-attention has been used to augment Transformer architectures with attention to the longer-horizon past (Rae et al., 2020; Dai et al., 2019) and to produce architectures that write to and/or read from ﬁxed-size arrays or memories (Santoro et al., 2018; Goyal et al., 2021), all…”
Can language models learn from explanations in context? (2022, background): “Thus, explanations may be more useful in future models which are better able to handle longer contexts (Press et al., 2021; Rae et al., 2019).”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “…robust and hold across model sizes (20M to 1.9B parameters), across compute scales (10 to 10 5 accelerator hours), across datasets (LM1B, C4, PG19 [22]), across hardware platforms (TPUv2, TPUv3, TPUv4 and V100), across multiple Transformer codebases using default conﬁgurations (Tensor2Tensor,…”
Scalable Adaptive Computation for Iterative Generation (2022, background): “While prior work has explored more dynamic and input-decoupled computation, e.g., networks with auxiliary memory (Dai et al., 2019; Rae et al., 2019) and global units (Zaheer et al., 2020; Burtsev et al., 2020; Jaegle et al., 2021b;a), general architectures that leverage adaptive computation to…”
Efficient Transformers: A Survey (2020, methodology): “…2020) O ( N log N ) LP Routing Transformer (Roy et al., 2020) O ( N N ) LP Axial Transformer (Ho et al., 2019) O ( N √ N ) FP Compressive Transformer (Rae et al., 2020) O ( N 2 ) RC Sinkhorn Transformer (Tay et al., 2020b) O ( B 2 ) LP Longformer (Beltagy et al., 2020) O ( n ( k + m )) FP+M ETC…”
ProGen: Language Modeling for Protein Generation (2020, background): “As a point of reference, state-of-the-art unidirectional language models for English Wikipedia achieve perplexities that range from 10 to 17 depending on model size (between 257 M and 8 .”
ETC: Encoding Long and Structured Inputs in Transformers (2020, methodology): “The Compressive Transformer (Rae et al., 2019) integrates this idea into Transformer-XL by compressing tokens in the input that are far away.”
AudioGen: Textually Guided Audio Generation (2022, background): “Modeling such long sequences is a notoriously difﬁcult problem (Rae et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020).”