Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks

By Xizhou Zhu, Jinguo Zhu, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang, Jifeng Dai. 2021

162 citations recorded by the source corpus.

Biological intelligence systems of animals perceive the world by integrating information in different modalities and processing simultaneously for various tasks. In contrast, current machine learning research follows a task-specific paradigm, leading to inefficient collaboration between tasks and high marginal costs of developing perception models for new tasks. In this paper, we present a generic perception architecture named Uni-Perceiver, which processes a variety of modalities and tasks with unified modeling and shared parameters. Specifically, Uni-Perceiver encodes different task inputs and targets from arbitrary modalities into a unified representation space with a modality-agnostic Transformer encoder and lightweight modality-specific tokenizers. Different perception tasks are modeled as the same formulation, that is, finding the maximum likelihood target for each input through the similarity of their representations. The model is pre-trained on several uni-modal and multi-modal tasks, and evaluated on a variety of downstream tasks, including novel tasks that did not appear in the pre-training stage. Results show that our pre-trained model without any tuning can achieve reasonable performance even on novel tasks. The performance can be improved to a level close to state-of-the-art methods by conducting prompt tuning on 1% of downstream task data. Full-data fine-tuning further delivers results on par with or better than state-of-the-art results. Code and pre-trained weights shall be released.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Concurrent works [17, 58, 119, 124, 154] also propose to formulate numerous vision tasks as text generation problems.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Following Uni-Perceiver [61], we evaluate our model on the 6 tasks of the GLUE benchmark, including single-sentence classiﬁcation and sentence pair classiﬁcation.”
Images Speak in Images: A Generalist Painter for In-Context Visual Learning (2022, background): “Vision Generalist Due to the general modeling capability of Transformer, there are some efforts to unify different tasks in vision domains, resulting in several vision generalists [9, 10, 27, 35, 55].”
YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (2024, methodology): “…of parameters utilization. in the field of deep learning have mainly focused on how to develop more powerful system architectures and learning methods, such as CNNs [21–23, 42, 55, 71, 72], Trans-formers [8, 9, 40, 41, 60, 69, 70], Perceivers [26, 26, 32, 52, 56, 81, 81], and Mambas [17, 38, 80].”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, unknown): “The pursuit of generalist models [74, 33, 62], which aim to handle a wide range of tasks using a shared architecture and parameters, has been a long-standing goal in the machine learning community.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, background): “Alternatively, image locations can be encoded as special tokens in the input/output text (Yang et al., 2021; Yao et al., 2022; Zhu et al., 2022a) following Chen et al. (2022b).”
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (2023, background): “…2021b; Lin et al., 2021; Kim et al., 2021; Dou et al., 2022; Zeng et al., 2021; Li et al., 2021a, 2022), especially in the development of multi-task generalist models (Hu and Singh, 2021; Singh et al., 2022; Zhu et al., 2022; Yu et al., 2022; Wang et al., 2022a; Lu et al., 2022a; Bai et al., 2022).”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (2023, background): “Concurrently, other research has been dedicated to the development of generalist models [19, 20, 22, 21], which permit a model to simultaneously input images and text, eliminating the necessity for a projection process.”
Continual Self-Supervised Learning: Towards Universal Multi-Modal Medical Data Representation Learning (2023, unknown): “We craft a universal architecture [59] in line with the aim of universal multi-modal SSL.”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “For example, using the multi-modal pre-trained weights from [86], ViTAdapter-BF yields an average result of 43.”
Visual Instruction Tuning with Polite Flamingo (2023, unknown): “Prior works towards this goal ( e.g, OFA (Wang et al. 2022a), Unified-IO (Lu et al. 2022), Uni-Perceiver (Zhu et al. 2022)) have focused on training multi-modal transformers via multi-task learning, but they lack the generalization ability to unseen tasks or instructions, and they are not capable…”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, background): “…2021; Li et al., 2021b; Lin et al., 2021; Kim et al., 2021; Dou et al., 2022; Li et al., 2021a, 2022), especially in the development of multi-task generalist models (Hu and Singh, 2021; Singh et al., 2022; Zhu et al., 2022; Yu et al., 2022; Wang et al., 2022a; Lu et al., 2022a; Bai et al., 2022).”