Perceiver: General Perception with Iterative Attention

By Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, João Carreira. 2021

1,523 citations recorded by the source corpus.

Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision models. These priors introduce helpful inductive biases, but also lock models to individual modalities. In this paper we introduce the Perceiver - a model that builds upon Transformers and hence makes few architectural assumptions about the relationship between its inputs, but that also scales to hundreds of thousands of inputs, like ConvNets. The model leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs. We show that this architecture is competitive with or outperforms strong, specialized models on classification tasks across various modalities: images, point clouds, audio, video, and video+audio. The Perceiver obtains performance comparable to ResNet-50 and ViT on ImageNet without 2D convolutions by directly attending to 50,000 pixels. It is also competitive in all modalities in AudioSet.

Subfields: Artificial Intelligence, Biophysics, Cognitive Neuroscience, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Cognitive Neuroscience

In Model Architectures

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “R d×d ǫ , W (i) Q ∈ R d×dτ & W (i) K ∈ R d×dτ are learnable projection matrices [35, 94].”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “We have incorporated several minor improvements to the training protocol reported in [35] and we use it for all experiments.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “Perceiver Resampler: from varying-size large feature maps to few visual tokens.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[105] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira.”
Scalable Adaptive Computation for Iterative Generation (2022, methodology): “…computation, e.g., networks with auxiliary memory (Dai et al., 2019; Rae et al., 2019) and global units (Zaheer et al., 2020; Burtsev et al., 2020; Jaegle et al., 2021b;a), general architectures that leverage adaptive computation to effectively scale to tasks with large input and output spaces…”
Accurate prediction of protein structures and interactions using a 3-track neural network (2021, positive): “To enable the network to focus on the most relevant sequence information for each region while keeping access to the full multiple sequence alignment in a more memory efficient way, we experimented with the Perceiver architecture (11), updating smaller seed MSAs (up to 100 sequences) with extra sequ”
Transformers in Vision: A Survey (2021, background): “Inspired by the biological systems that can process information from a diverse range of modalities, Perceiver model [90] aims to learn a uniied model that can process any given input modality without making domain-speciic architectural assumptions.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “[7] and [58] propose a simple framework that can process information from multiple modalities with a uniform byte-sequence representation.”
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, background): “Perceiver [19] leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs.”
Coordination Among Neural Modules Through a Shared Global Workspace (2021, background): “Concurrent to our work, (Jaegle et al., 2021) also introduced the idea of using a latent bottleneck for addressing quadratic complexity by learning a bottleneck but there are important differences.”
XCiT: Cross-Covariance Image Transformers (2021, background): “Other efﬁcient attention methods rely on local attention and adding a small number of global tokens, thus allowing interaction among all tokens only by hopping through the global tokens [1, 5, 34, 80].”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “To do this, we build on the Perceiver family of attention architectures (Jaegle et al., 2021; 2022), which have demonstrated excel-a Causal masking is used in both cross-and self-attention to maintain end-to-end autoregressive ordering. lent performance on a wide range of large-context domains.”
Query-Centric Trajectory Prediction (2023, methodology): “Then, we transform each polar coordinate into Fourier features [22, 35, 44] to facilitate learning high-frequency signals.”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, methodology): “It should be noted that some attention methods below, such as Perceiver [76] and Criss-Cross [107, 191], are transformer types [165], but are customized for multi-modal tasks by including text, audio, and image.”
Improved Transformer for High-Resolution GANs (2021, background): “Some other works [7, 21, 22] avoid directly applying standard self-attention to the input pixels, and perform attention between the input and a small set of latent units.”