OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

By Peng Wang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang. 2022

1,080 citations recorded by the source corpus.

In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision&language models that rely on extremely large cross-modal datasets, OFA is pretrained on only 20M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a series of cross-modal tasks while attaining highly competitive performances on uni-modal tasks. Our further analysis indicates that OFA can also effectively transfer to unseen tasks and unseen domains. Our code and models are publicly available at https://github.com/OFA-Sys/OFA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “With the vast advancement of the Transformer architecture [35, 189] in both CV and NLP, there is a trend to pre-train large-scale unified vision-language (VL) models [1, 27, 105, 196, 197, 200] to perform vision grounded text generation tasks, such as image captioning and visual question answering (”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “A line of works [36, 46, 50] train both visual and textual encoders from scratch, leading to a high computation cost.”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “…et al., 2021; Jia et al., 2021), the fusion-encoder architecture (Tan & Bansal, 2019; Li et al., 2021), the encoder-decoder architecture (Cho et al., 2021; Wang et al., 2021b; Chen et al., 2022b), and more recently, the unified transformer architecture (Li et al., 2022; Wang et al., 2022b).”
Generalized Decoding for Pixel, Image, and Language (2022, background): “Recently, a number of works aim to reformulate the tasks into a unified sequential decoding process [7, 38, 54, 76, 85].”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, neutral): “To reduce this discrepancy, recent approaches (Cho et al., 2021; Wang et al., 2021b; Yang et al., 2021b; Wang et al., 2022b) have attempted to design unified generative models for pre-training, as most VL tasks can be cast as generation problems.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “GIT2 (Wang et al., 2022a) has demonstrated strong performance without the OCR input, while PaLI-17B shows the superiority of levaraging a specialized OCR system for a better recipe to solve these tasks.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “For a more comprehensive comparison of human datasets, we refer interested readers to [217].”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (2023, methodology): “For instance, OFA [27] devised a unified sequence-to-sequence decoding architecture applicable to language and object detection tasks.”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, methodology): “Beyond image captioning pre-training task, some research [15,55,79] attempts to formulate more vision tasks in a unified sequence-to-sequence learning paradigm, including object detection and image segmentation.”
Planning-oriented Autonomous Driving (2022, background): “This is a popular practice in many domains, including general vision [46, 51, 61], autonomous driving2 [8, 34, 57, 59], such as Transfuser [13], BEV-”
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (2023, methodology): “, 2023), OFA (Wang et al., 2022) and Shikra (Chen et al.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, background): “SimVLM [17] and OFA [18] train encoder-decoder models with generative targets and show competitive performances on a series of multimodal tasks.”