MIMIC-IT: Multi-Modal In-Context Instruction Tuning
By Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, Ziwei Liu. 2023
310 citations recorded by the source corpus.
High-quality instructions and responses are essential for the zero-shot performance of large language models on interactive natural language tasks. For interactive vision-language tasks involving intricate visual scenes, a large quantity of diverse and creative instruction-response pairs should be imperative to tune vision-language models (VLMs). Nevertheless, the current availability of vision-language instruction-response pairs in terms of quantity, diversity, and creativity remains limited, posing challenges to the generalization of interactive VLMs. Here we present MultI-Modal In-Context Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos. Each pair is accompanied by multi-modal in-context information, forming conversational contexts aimed at empowering VLMs in perception, reasoning, and planning. The instruction-response collection process, dubbed as Syphus, is scaled using an automatic annotation pipeline that combines human expertise with GPT's capabilities. Using the MIMIC-IT dataset, we train a large VLM named Otter. Based on extensive evaluations conducted on vision-language benchmarks, it has been observed that Otter demonstrates remarkable proficiency in multi-modal perception, reasoning, and in-context learning. Human evaluation reveals it effectively aligns with the user's intentions. We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Language and Linguistics, Alignment & RLHF, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Yu Qiao
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
In Language and Linguistics
In Multimodal Learning
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, background): “Recent works further demonstrate improved performance by scaling up the pretraining data [3, 13, 53], instruction-following data [13, 17, 28, 57], visual encoders [3], or language models [38], respectively.”
- Aligning Large Multimodal Models with Factually Augmented RLHF (2023, background): “2 2 For a comprehensive breakdown of each dataset’s influence, refer to Appendix A.1.”
- V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “The type of projection module varies across different models. including options like Resampler [1, 22, 47], QFormer [8, 23], and linear layer [5, 28].”
- OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models (2023, methodology): “[20] fine-tuned OpenFlamingo on MIMIC-IT [19], a multi-image/video instruction following dataset, creating Otter, a”
- InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Inspired by recent vision-centric dialogue systems [10, 77, 78], we integrate our pretrained ViCLIP (with InternVid) into VideoChat [10] to show how our data and model can empower multimodal dialogue methods with effective video modeling capability.”
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (2023, methodology): “In the second stage, we finetune the pretrained model with a smaller but high-quality image-text dataset with a designed conversational template to enhance generation reliability and usability.”
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, methodology): “Approaches like MiniGPT4 [61], LLaVA [24], MIMIC-IT [16], VideoChat [18], and Valley [28] obtain detailed visual descriptions and build image-centric or video-centric conversation data from LLMs.”
- CogAgent: A Visual Language Model for GUI Agents (2023, unknown): “On the POPE-adversarial evaluation, our model attained a score of 85.9, demonstrating superior handling of Vicuna-13B 22.4 - Otter [17] MPT-7B 24.7 - MiniGPT4 [44] Vicuna-13B 24.4 70.4 InstructBLIP [9] Vicuna-13B 25.6 77.3 LLaVA [21] LLaMA2-7B 28.1 66.3 LLaMA-Adapter v2 [14] LLaMA-7B 31.4 -…”
- An Embodied Generalist Agent in 3D World (2023, unknown): “To address this issue, existing methods provide bounding boxes (Liu et al., 2023b) and add dense captions (Li et al., 2023a; Liu et al., 2023a) as image descriptions or directly use off-the-shelf large vision-language models (LVLM) (Zhu et al., 2023a; Luo et al., 2023) to help collect such data.”
- MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “They extend LLMs to tackle a diverse range of tasks and domains, including the mainstream 2D images [35, 15, 1, 31] and other modalities, such as 3D point clouds [23, 60, 27], audio [24, 52], and video [65, 6].”
- MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, methodology): “Typically, increasing the model size (Zhang et al., 2023a; Bai et al., 2023b) and dataset scale (Zhang et al., 2023c; Zhao et al., 2023a; Chen et al., 2023d) can improve model performance.”