Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

By Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, Aniruddha Kembhavi. Computer Vision and Pattern Recognition, 2023

353 citations recorded by the source corpus.

We present Unified-IO 2,the. first autoregressive multi-modal model that is capable of understanding and generating image, text, audio, and action. To unify different modalities, we tokenize inputs and outputs - images, text, audio, action, bounding boxes etc., into a shared semantic space and then process them with a single encoder-decoder transformer model. Since training with such diverse modalities is challenging, we propose various architectural improvements to stabilize model training. We train our model from scratch on a large multimodal pre-training corpus from diverse sources with a multimodal mixture of denoisers objective. To learn an expansive set of skills, such as following multimodal instructions, we construct and. finetune on an ensemble of 120 datasets with prompts and augmentations. With a single unified model, Unified-io 2 achieves state-of-the-art performance on the GRIT benchmark and strong results in more than 35 benchmarks, including image generation and understanding, natural language understanding, video and audio understanding, and robotic manipulation. We release all our models to the research community.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6c64ddd2190909de2c680dd18abc9b92e80c39f9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

Ranjay Krishna

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, background): “As multimodal language models (LMs) [35, 42, 28, 27, 1, 3, 49, 7, 31, 41, 6, 5] have begun to mature, we now expect them to solve tasks like the ones mentioned above, i.e., ones where people draw intermediate sketches to simplify reasoning.”
What matters when building vision-language models? (2024, neutral): “TabMWP (Lu et al., 2023) 22,729 23,059 1,948,166 2.49% TAT-QA (Zhu et al., 2021) 2,199 13,215 283,776 2.18% HiTab (Cheng et al., 2022) 2,500 7,782 351,299 1.80% MultiHiertt (Zhao et al., 2022) 7,619 7,830 267,615 1.37% FinQA (Chen et al., 2021) 5,276 6,251 242,561 0.99% WikiSQL (Zhong et al., 2017)…”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, neutral): “We build our CoT-VLA system that leverages visual chain-of-thought reasoning upon recent advances in unified multimodal foundation models that can understand and generate text and images [36, 52, 55, 61, 63].”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, neutral): “In recent years, an increasing number of studies (Ge et al., 2024; Wu et al., 2023b; Tang et al., 2024; Ye et al., 2024a) have focused on unified multimodal language models capable of both comprehension and generation.”
BLINK: Multimodal Large Language Models Can See but Not Perceive (2024, background): “…in the modern era of large language models (LLMs), we, as a community, have focused less on such perceptual tasks, and instead have developed new tasks, mostly expressed in natural language, emphasizing the vision-language connection learned by multimodal LLMs [3,6,18,20,24,26,50,51,56,62,71,77].”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “In parallel, the field of computer vision has been striving to develop large autoregressive or world models [42, 68, 5], aiming to emulate their impressive scalability and generalizability.”
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, neutral): “Some work [15, 14, 13, 22, 23] perform multimodal autoregression with a unified next-word-prediction objective through using a discrete visual tokenizer.”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, methodology): “We note that this design is not an ultimate design for multimodal foundation models, where a unified vocabulary is established between language and vision [Lu et al. 2023; Team et al. 2023].”
Emerging Properties in Unified Multimodal Pretraining (2025, neutral): “Typical design choices for unified multi-modal generation and understanding models include: Quantized AR. Autoregressive visual generation [11, 48, 59, 70, 79, 83–85, 90 ] Unfortunately, the visual generation quality of autoregressive models is empirically inferior to diffusion-based models.”
Chameleon: Mixed-Modal Early-Fusion Foundation Models (2024, neutral): “We note that similar attempts at stabilizing multi-modal trainings were concurrently discovered in (Lu et al., 2023).”
AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning (2025, unknown): “…rapid evolution of multi-modal research, particularly in the audio-visual domain, has produced AV-LLMs [Cheng et al. 2024, Hurst et al. 2024a, Team et al. 2023, Wang et al. 2024b] and OLMs [Liu et al. 2025, Lu et al. 2024a, Xu et al. 2025b, Li et al. 2025, Team 2025] with remarkable capabilities.”