SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
By Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, Ying Shan. arXiv.org, 2024
339 citations recorded by the source corpus.
The rapid evolution of multimodal foundation model has demonstrated significant progresses in vision-language understanding and generation, e.g., our previous work SEED-LLaMA. However, there remains a gap between its capability and the real-world applicability, primarily due to the model's limited capacity to effectively respond to various user instructions and interact with diverse visual data. In this work, we focus on bridging this gap through integrating two enhanced features: (1) comprehending images of arbitrary sizes and ratios, and (2) enabling multi-granularity image generation. We present a unified and versatile foundation model, namely, SEED-X, which is able to model multi-granularity visual semantics for comprehension and generation tasks. Besides the competitive results on public benchmarks, SEED-X demonstrates its effectiveness in handling real-world applications across various domains after instruction tuning. We hope that our work will inspire future research into what can be achieved by versatile multimodal foundation models in real-world applications. The models, codes, and datasets are released in https://github.com/AILab-CVC/SEED-X.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/ef03c907ee24e2e6c0f2d24639551c82862d1080
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Xinlong Wang
- Xiaosong Zhang
- Zhengxiong Luo
- Yueze Wang
- Quan Sun
- Yufeng Cui
- Jinsheng Wang
- Fan Zhang
Citations in the corpus
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “Recent advancements have explored extending MLLMs with pretrained diffusion models to facilitate image generation [26, 28, 34, 72, 73].”
- Emerging Properties in Unified Multimodal Pretraining (2025, neutral): “LLM backbone combined with an external diffusion module [18, 23, 57, 69, 73].”
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “Zero-shot generalization [55] refers to the ability of a model, particularly a Large Language Model, to perform tasks that it has not been explicitly trained on.”
- Emu3: Next-Token Prediction is All You Need (2024, background): “There have been early efforts to unify vision understanding and generation [82, 97, 25, 21], exploring various generative objectives on image and text data.”