Perceiver IO: A General Architecture for Structured Inputs & Outputs

By Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Andrew Brock, Evan Shelhamer, Olivier J. H'enaff and 4 more. 2021

871 citations recorded by the source corpus.

A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied beyond a small set of stereotyped settings, as they bake in domain&task assumptions or scale poorly to large inputs or outputs. In this work, we propose Perceiver IO, a general-purpose architecture that handles data from arbitrary settings while scaling linearly with the size of inputs and outputs. Our model augments the Perceiver with a flexible querying mechanism that enables outputs of various sizes and semantics, doing away with the need for task-specific architecture engineering. The same architecture achieves strong results on tasks spanning natural language and visual understanding, multi-task and multi-modal reasoning, and StarCraft II. As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

Subfields: Artificial Intelligence, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

Citations in the corpus

High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “We turn DMs into more flexible conditional image generators by augmenting their underlying UNet backbone with the cross-attention mechanism [94], which is effective for learning attention-based models of various input modalities [34,35].”
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “…Zhang et al., 2021), efficient attention (Tay et al., 2020c; Kitaev et al., 2020; Choromanski et al., 2020; Tay et al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al., 2019), alternative non Transformer-based architectures (Tolstikhin et al., 2021; Tay et al.,…”
Multiview Transformers for Video Recognition (2022, methodology): “We also note that “bottleneck” tokens have also been used by [31, 49].”
Multiview Compressive Coding for 3D Reconstruction (2023, background): “Another approach is cross-attention (cross-attn), where the encoded input R only serves as keys / values but not as queries to a transformer, e.g ., in Perceiver models [22, 23].”
The Efficiency Misnomer (2021, neutral): “Token-Free Models Token-free models (Xue et al., 2021; Tay et al., 2021c; Jaegle et al., 2021) get rid of the large subword vocabulary by modeling at the character or byte level.”
Scalable Adaptive Computation for Iterative Generation (2022, methodology): “…computation, e.g., networks with auxiliary memory (Dai et al., 2019; Rae et al., 2019) and global units (Zaheer et al., 2020; Burtsev et al., 2020; Jaegle et al., 2021b;a), general architectures that leverage adaptive computation to effectively scale to tasks with large input and output spaces…”
Efficient Transformers: A Survey (2020, methodology): “…(Zhu et al., 2021) O ( kN ) FP + LR Poolingformer (Zhang et al., 2021) O ( N ) FP+M Nystr¨omformer (Xiong et al., 2021b) O ( kN ) M+DS Perceiver (Jaegle et al., 2021) O ( kN ) M+DS Clusterformer (Wang et al., 2020b) O ( N log N ) LP Luna (Ma et al., 2021) O ( kN ) M TokenLearner (Ryoo et…”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “PerceiverIO [36] proposed a general architecture whose output is adapted to different modalities using specific learned tokens, and whose computation is reduced using a small number of latent tokens.”
Transformers in Vision: A Survey (2021, background): “A recent improvement called Perceiver IO [89] aims to learn models with both lexible inputs as well as arbitrary sized outputs.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “[7] and [58] propose a simple framework that can process information from multiple modalities with a uniform byte-sequence representation.”
A Generalist Agent (2022, background): “Perceiver IO (Jaegle et al., 2021) uses a transformer-derived architecture specialized for very long sequences, to model any modality as a sequence of bytes.”
Language Models are General-Purpose Interfaces (2022, background): “Jaegle et al. (2022) introduce Perceiver IO, a general architecture across multiple domains including language/visual understanding, multimodal and symbolic representations for games.”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, background): “This strategy leads to an architecture (Figure 1) with complexity O ( MN )+ O ( LN 2 ) due to the cross-attention and the latent self-attention stack, respectively (Jaegle et al., 2022).”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, neutral): “The computational complexity of their attention layers prohibits na¨ıvely scaling them to higher resolutions and several works proposed alternatives (Zhu et al., 2021; Liu et al., 2021; Li et al., 2022c; Chu et al., 2021; Liu et al., 2022a; 2023; Cai et al., 2023; Jaegle et al., 2022).”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “These elements are also encoded using the ViT or AST, but we then use a per-ceiver resampler [5, 86], see Table 7 for hyperparameters, to further compress the features into a smaller number of tokens (32 for images and 16 for audio).”
Images Speak in Images: A Generalist Painter for In-Context Visual Learning (2022, background): “Perceiver [22] and Perceiver-IO [21] are the first attempts to use the exact same Transformer architecture in different domains, such as natural language and visual understanding processing, StarCraft II, and multi-modal domains.”
Webly Supervised Concept Expansion for General Purpose Vision Models (2022, background): “Recent work shows promise in this regard [30], and raises the potential of transferring concepts from web data to an even wider range of tasks.”
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks (2022, background): “Moreover, allowing all inputs to self-attend to each other early on the encoding process reduces complexity and the need to handcraft architectures to address the scaling of computation for transformers with the increase in the input sequence length [48].”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, neutral): “Models like Flamingo [4], PerceiverIO [33], BEiT-3 [48], GPT-4 [42] show remarkable versatility, dealing with diverse data sources and tackling new tasks by observing only a handful of examples.”