MIMIC-IT: Multi-Modal In-Context Instruction Tuning

By Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, Ziwei Liu. 2023

310 citations recorded by the source corpus.

High-quality instructions and responses are essential for the zero-shot performance of large language models on interactive natural language tasks. For interactive vision-language tasks involving intricate visual scenes, a large quantity of diverse and creative instruction-response pairs should be imperative to tune vision-language models (VLMs). Nevertheless, the current availability of vision-language instruction-response pairs in terms of quantity, diversity, and creativity remains limited, posing challenges to the generalization of interactive VLMs. Here we present MultI-Modal In-Context Instruction Tuning (MIMIC-IT), a dataset comprising 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos. Each pair is accompanied by multi-modal in-context information, forming conversational contexts aimed at empowering VLMs in perception, reasoning, and planning. The instruction-response collection process, dubbed as Syphus, is scaled using an automatic annotation pipeline that combines human expertise with GPT's capabilities. Using the MIMIC-IT dataset, we train a large VLM named Otter. Based on extensive evaluations conducted on vision-language benchmarks, it has been observed that Otter demonstrates remarkable proficiency in multi-modal perception, reasoning, and in-context learning. Human evaluation reveals it effectively aligns with the user's intentions. We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Language and Linguistics, Alignment & RLHF, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yu Qiao
  2. Jianfeng Gao
  3. Percy Liang
  4. Mohit Bansal
  5. Quoc V. Le
  6. Ziwei Liu
  7. Yoshua Bengio
  8. Trevor Darrell

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Yu Qiao
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Trevor Darrell
  6. Ming-Hsuan Yang
  7. Jitendra Malik
  8. Andrew Zisserman

In Language and Linguistics

  1. Yong Jae Lee
  2. Limin Wang
  3. Chunyuan Li
  4. Haotian Liu
  5. Sheng Shen
  6. Shiori Sagawa
  7. Pang Wei Koh
  8. Jack Hessel

In Multimodal Learning

  1. Yu Qiao
  2. Saining Xie
  3. Ping Luo
  4. Peng Gao
  5. Penghao Wu
  6. Bo Li
  7. Yuanhan Zhang
  8. Renrui Zhang

Citations in the corpus