Generative Multimodal Models are In-Context Learners

By Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang and 1 more. Computer Vision and Pattern Recognition, 2023

522 citations recorded by the source corpus.

The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4b1b5e219fb41a7413599c3b2ca6a7fdf045d1a5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Li
2. Renrui Zhang
3. Yuanhan Zhang
4. Zejun Ma
5. Chunyuan Li
6. Wei Li
7. Jinyu Li
8. Tara N. Sainath

In Computer Vision and Pattern Recognition

1. Renrui Zhang
2. Yuanhan Zhang
3. Bo Li
4. Yueze Wang
5. Jingjing Liu
6. Zejun Ma
7. Tiejun Huang
8. Yufeng Cui

Citations in the corpus

• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, background): “The ICL performance is typically considered to evaluate multimodal pre-training, which has been adopted in several known LMMs, such as OpenFlamingo [2], IDEFICS series [22, 23], VILA [33] and MM1 [41], Emu2 [51].”
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, unknown): “We compare image manipulation with previous MLLMs including Emu2-Gen [24], Gemini [33], MGIE [33] and Mini-Gemini [34].”
• Emu3: Next-Token Prediction is All You Need (2024, methodology): “Emu and Emu2 [82, 81] introduce a unified autoregressive objective: predicting the next multimodal element, by regressing visual embeddings or classifying textual tokens.”
• CogVLM: Visual Expert for Pretrained Language Models (2023, unknown): “…ablation studies on several structure variants and tuning strategies, including: 1) tuning only the MLP Adapter layer; 2) tuning all LLM parameters and the Adapter without adding visual expert; 3) only adding visual expert at every 4th LLM layer; and 4) only add visual expert to FFNs at all layers.”
• Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, unknown): “Additionally, the integration of LLMs with specific tools for visual tasks [34, 35] highlights their adaptability and potential for broad application, underscoring the utility of LLMs in extending beyond traditional text-based processing to include multimodal interactions.”
• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, unknown): “Recent advancements have explored extending MLLMs with pretrained diffusion models to facilitate image generation [26, 28, 34, 72, 73].”
• Emerging Properties in Unified Multimodal Pretraining (2025, unknown): “Unlike some prior approaches [18, 57, 69, 73] that introduce bottleneck connectors between generation and understanding modules, our design enables long-context interaction between multimodal understanding and generation through shared self-attention operations.”
• What matters when building vision-language models? (2024, unknown): “Although EVA-CLIP-5B (Sun et al., 2023) is ten times bigger in parameter counts than SigLIP-SO400M (Zhai et al., 2023), we obtain similar performance across 4 benchmarks, suggesting that EVA-CLIP-5B could be heavily under-trained, and we acknowledge that the open VLM community is missing a large wel”
• DanceGRPO: Unleashing GRPO on Visual Generation (2025, unknown): “We also hope our work offers insights into its potential to unify optimization paradigms across different modalities (e.g. image and text) [55, 56].”