Show-o: One Single Transformer to Unify Multimodal Understanding and Generation

By Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, Mike Zheng Shou. International Conference on Learning Representations, 2024

730 citations recorded by the source corpus.

We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model. Code and models are released at https://github.com/showlab/Show-o.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/9dc337bd18bc0201839942931b12aff8ec1c93f5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chong Ruan
2. Zhenda Xie

Citations in the corpus

• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, positive): “Other approaches [74, 82, 83, 91] employ a single trans-former to unify both multimodal understanding and generation tasks, which improves instruction-following for visual generation, unlocks potential emergent abilities, and reduces model redundancy.”
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, neutral): “Some work [15, 14, 13, 22, 23] perform multimodal autoregression with a unified next-word-prediction objective through using a discrete visual tokenizer.”
• Emu3: Next-Token Prediction is All You Need (2024, methodology): “The results of other methods in the GenEval are sourced from [26, 92, 108, 22].”
• CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, neutral): “Recent advances in unified multimodal models [55, 59, 63, 73] suggest promising directions for improvements.”
• Emerging Properties in Unified Multimodal Pretraining (2025, neutral): “The field of unified multimodal understanding and generation has witnessed a surge in interest, with numerous research projects [57, 66, 68, 70, 73, 79, 83, 89] demonstrate promising results in jointly optimizing generation and understanding benchmarks with a crafted unified architecture.”