MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

By Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, Baobao Chang. 2023

213 citations recorded by the source corpus.

Since the resurgence of deep learning, vision-language models (VLMs) enhanced by large language models (LLMs) have grown exponentially in popularity. However, while LLMs can utilize extensive background knowledge and task information with in-context learning, most VLMs still struggle with understanding complex multi-modal prompts with multiple images, making VLMs less effective in downstream vision-language tasks. In this paper, we address the limitation above by 1) introducing vision-language Model with Multi-Modal In-Context Learning(MMICL), a new approach to allow the VLM to deal with multi-modal inputs efficiently; 2) proposing a novel context scheme to augment the in-context learning ability of the VLM; 3) constructing the Multi-modal In-Context Learning (MIC) dataset, designed to enhance the VLM's ability to understand complex multi-modal prompts. Our experiments confirm that MMICL achieves new state-of-the-art zero-shot performance on a wide range of general vision-language tasks, especially for complex benchmarks, including MME and MMBench. Our analysis demonstrates that MMICL effectively tackles the challenge of complex multi-modal prompt understanding and emerges the impressive ICL ability. Furthermore, we observe that MMICL successfully alleviates language bias in VLMs, a common issue for VLMs that often leads to hallucination when faced with extensive textual context. Our code, dataset, dataset tool, and model are available at https://github.com/PKUnlp-icler/MIC

Subfields: Artificial Intelligence, Geography, Planning and Development, Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongsheng Li
2. Yu Qiao
3. Peng Gao
4. Renrui Zhang
5. Jiaming Han
6. Dahua Lin
7. Yuhang Cao
8. Yuhang Zang

In Prompting & Reasoning

1. Huan Sun
2. Fei Huang
3. Yu Qiao
4. Peng Gao
5. Ziyi Lin
6. Renrui Zhang
7. Ge Zhang
8. Hongsheng Li

In Multimodal Learning

1. Peng Gao
2. Hongsheng Li
3. Yu Qiao
4. Renrui Zhang
5. Huan Sun
6. Yuhang Zang
7. Fei Huang
8. Qi Qian

Citations in the corpus

• Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “However, there is severe information loss in the captioning process; thus, using the captions (as opposed to vision features) may suffer from a lack of mutual synergy in the representation space of different modalities.”
• MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, methodology): “This area has been explored in depth by models such as Flamingo [2] and OpenFlamingo [4], Otter [31], M3IT [33], MetaVL [51], Sparkles [25], and MMICL [83].”
• InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, background): “A series of studies [2, 5, 9, 10, 20, 21, 42, 46, 49, 62, 64, 80, 86, 92, 97, 98, 100] have explored further improve the MLLM in different dimensions, such as instruction tuning [11, 49, 98], efficient fine-tuning [33], high-resolution image inputs [6, 82, 83], hallucination mitigation [34, 87,…”
• EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, background): “…[5], [6], [7], has shown great success in the natural image domain, as they not only possess robust language interaction capabilities but also exhibit impressive multitask reasoning skills in real-world scenarios [8], compared to smaller and domain-specific models tailored for particular tasks.”
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, unknown): “…capabilities (Liu et al., 2023c,b, 2024a; Li et al., 2024b; Dai et al., 2023; Zhu et al., 2023; Zhang et al., 2023; Gao et al., 2023; Ye et al., 2023a,b; Zhao et al., 2023; Li et al., 2023; Monajatipoor et al., 2023; Zhao et al., 2024; Li et al., 2024c; Lin et al., 2024; Zhang et al., 2024).”
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, unknown): “2”
• An Embodied Generalist Agent in 3D World (2023, unknown): “In particular, LEO directly produces motor commands without inductive bias ( e.g ., heatmap) that benefit previous methods, showcasing LEO’s considerable capacity for learning embodied actions.”
• VILA: On Pre-training for Visual Language Models (2023, unknown): “Some work freezes the LLM and train auxiliary components [5, 32, 33, 58], others fine-tune the LLM to enable visual capabilities [13, 19, 66].”
• SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, background): “To further expand their capacity ceiling, visual images are also introduced as inputs to develop powerful multi-modal large language models (MLLMs) (Zhang et al., 2023a; Li et al., 2023d; Liu et al., 2023d; Zhu et al., 2023; Zhao et al., 2023).”
• mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “…methods are primarily based on interleaved image-text web data for pre-training (Laurençon et al., 2023; Laurençon et al., 2024) to extend multi-image capabilities or focused on the in-context abilities (Alayrac et al., 2022; Awadalla et al., 2023; Zhao et al., 2023) within multi-image scenarios.”
• Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, unknown): “Concurrently, to maintain OpenFlamingo’s in-context learning capacity, we retrieve in-context examples for each triplet, which often lack correlated context, such as a visual question-answer data sample in VQAv2 [3].”