General-purpose, long-context autoregressive modeling with Perceiver AR

By Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon and 5 more. 2022

87 citations recorded by the source corpus.

Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. Perceiver AR can directly attend to over a hundred thousand tokens, enabling practical long-context density estimation without the need for hand-crafted sparsity patterns or memory mechanisms. When trained on images or music, Perceiver AR generates outputs with clear long-term coherence and structure. Our architecture also obtains state-of-the-art likelihood on long-sequence benchmarks, including 64 x 64 ImageNet images and PG-19 books.

Subfields: Computer Vision and Pattern Recognition, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Large Language Models

In Model Architectures

Citations in the corpus

SoundStorm: Efficient Parallel Audio Generation (2023, neutral): “…approaches, or a combination thereof: i) efﬁcient attention mechanisms (Kitaev et al., 2020; Choromanski et al., 2021; Xiong et al., 2021; Hawthorne et al., 2022), ii) non-autoregressive, parallel decoding schemes (Gu et al., 2017; Ghazvininejad et al., 2019; Chang et al., 2022), iii)…”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “While several works have explored efficient alternatives to self-attention [39], [40], [41], another solution to this scaling problem is to work with mappings of the natural signals to a compact, discrete representation space.”
MusicLM: Generating Music From Text (2023, background): “PerceiverAR (Hawthorne et al., 2022a), on the other hand, proposes to model a sequence of SoundStream tokens autoregressively, achieving high-quality audio, but compromising the long-term temporal coherence.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “A known limitation is the slow generation speed, but this issue could be alleviated by future research on parallel sampling [78, 79, 80], reducing the number of latent variables [81], and improvements in hardware accelerators.”
Hungry Hungry Hippos: Towards Language Modeling with State Space Models (2022, methodology): “18.5 Performer (125M) [9] 26.8 Reformer (125M) [39] 26.0 Linear Attention (125M) [35] 25.6 Perceiver AR (358M) [30] 18.4 Transformer-XL (285M) [13] 18.4 Table 10 shows that the Hybrid H3 model is competitive with Transformers of the same size, as well as larger models such as the 358M Perceiver AR…”
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (2023, methodology): “As shown in Table 4, M EGA B YTE matches the state-of-the-art performance of PerceiverAR whilst using only half the compute.”
JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models (2023, methodology): “AR models, such as PerceiverAR [13], AudioGen [2], Musi-cLM [3], and Jukebox [14], predict audio tokens sequentially based on prior context.”