Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

By Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, Aniruddha Kembhavi. 2022

530 citations recorded by the source corpus.

We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing. Developing a single unified model for such a large variety of tasks poses unique challenges due to the heterogeneous inputs and outputs pertaining to each task, including RGB images, per-pixel maps, binary masks, bounding boxes, and language. We achieve this unification by homogenizing every supported input and output into a sequence of discrete vocabulary tokens. This common representation across all tasks allows us to train a single transformer-based architecture, jointly on over 90 diverse datasets in the vision and language fields. Unified-IO is the first model capable of performing all 7 tasks on the GRIT benchmark and produces strong results across 16 diverse benchmarks like NYUv2-Depth, ImageNet, VQA2.0, OK-VQA, Swig, VizWizGround, BoolQ, and SciTail, with no task-specific fine-tuning. Code and demos for Unified-IO are available at: https://unified-io.allenai.org.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, background): “However, recently several “generalist” models have been developed, such as UnifiedIO [Lu et al., 2022] and FLAVA [Singh et al.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “, 2022b) and a concurrent work (Lu et al., 2022), we resolve this by using a sufficiently general interface for all tasks considered: the model accepts as input an image and text string, and generates text as output.”
Inductive biases for deep learning of higher-level cognition (2020, neutral): “with efficient transfer learning) (Ravi and Larochelle, 2016; Wang et al., 2016; Finn et al., 2017; Cabi et al., 2019; Jang et al., 2022; Reed et al., 2022; Ahn et al., 2022; Brown et al., 2020; Alayrac et al., 2022; Borgeaud et al., 2022; Chowdhery et al., 2022; Sanh et al., 2021; Lu et al., 2022; ”
GLIPv2: Unifying Localization and Vision-Language Understanding (2022, background): “Recently, a general interest arises in building general-purpose vision systems [21, 24, 56, 42], also called vision foundation models [6, 57], that solve various vision tasks simultaneously, such as image classification [30], object detection [39], and Visual-Language (VL) understanding [3, 11, 27].”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, background): “Work on multi-modal large models capable of both processing visual modalities effectively and generating natural language fluently (Huang et al., 2023; Driess et al., 2023; Lu et al., 2022) would help overcome this limitation and build better grounded embodied agents.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “GATO [95] and Unified-IO [74] learn a single model to solve various decision-making tasks with different control interfaces.”
Transformers in Time-Series Analysis: A Tutorial (2022, background): “Apart from classical NLP tasks, examples from other areas include image classification [3], object detection and segmentation [4], image and language generation [5], sequential decision-making in reinforcement learning [6], multi-modal (text, speech, and image) data processing [7], and analysis of t”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Our work is also related to recent generalist vision models [10, 11, 60, 31, 43] where both architecture and loss functions are task-agnostic.”
An Embodied Generalist Agent in 3D World (2023, unknown): “Nonetheless, existing generalist models primarily thrive within 2D domains, lacking comprehension of the 3D physical environment that envelops human-level intelligence.”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “Transformers (Vaswani et al., 2017) have enabled task unification across many AI domains (Brown et al., 2020; Chen et al., 2022a;b; Lu et al., 2022; Wang et al., 2022c).”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “For cross-modal creation, early works generally tokenize the visual contents into discrete VQ codebooks (van den Oord et al., 2017; Wang et al., 2022; Sun et al., 2022; Lu et al., 2023; Diao et al., 2023; Yu et al., 2023a).”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “Following [123], we represent per-pixel labels, which include depth, surface normals, and binary segmentation masks, as RGB images that can be generated or encoded with our image generation and encoding abilities.”
Towards Generalist Biomedical AI (2023, background): “Recent work has also explored generalist models capable not only of performing many tasks, but also of processing many modalities [41].”
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning (2022, background): “Several recent studies (Cho et al., 2021; Wang et al., 2022a,c; Lu et al., 2022) also started to build a unified pre-training framework to handle a diverse set of cross-modal and unimodal tasks.”