Otter: A Multi-Modal Model With In-Context Instruction Tuning

By Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, Ziwei Liu. 2023

692 citations recorded by the source corpus.

Recent advances in Large Multimodal Models (LMMs) have unveiled great potential as visual assistants. However, most existing works focus on responding to individual instructions or using previous dialogues for contextual understanding. There is little discussion on employing both images and text as in-context examples to enhance the instruction following capability. To bridge this gap, we introduce the Otter model to leverage both textual and visual in-context examples for instruction tuning. Specifically, Otter builds upon Flamingo with Perceiver architecture, and has been instruction tuned for general purpose multi-modal assistant. Otter seamlessly processes multi-modal inputs, supporting modalities including text, multiple images, and dynamic video content. To support the training of Otter, we present the MIMIC-IT (MultI-Modal In-Context Instruction Tuning) dataset, which encompasses over 3 million multi-modal instruction-response pairs, including approximately 2.2 million unique instructions across a broad spectrum of images and videos. MIMIC-IT has been carefully curated to feature a diverse array of in-context examples for each entry. Comprehensive evaluations suggest that instruction tuning with these in-context examples substantially enhances model convergence and generalization capabilities. Notably, the extensive scenario coverage provided by the MIMIC-IT dataset empowers the Otter model to excel in tasks involving complex video and multi-image understanding.

Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Yu Qiao
  4. Percy Liang
  5. Quoc V. Le
  6. Yoshua Bengio
  7. Ping Luo
  8. Dacheng Tao

In Large Language Models

  1. Yu Qiao
  2. Ping Luo
  3. Maosong Sun
  4. Zhiyuan Liu
  5. Bo Li
  6. Yuanhan Zhang
  7. Peng Jin
  8. Haotian Liu

In Alignment & RLHF

  1. Rohan Taori
  2. Chelsea Finn

In Multimodal Learning

  1. Yu Qiao
  2. Hongsheng Li
  3. Ping Luo
  4. Peng Gao
  5. Renrui Zhang
  6. Pan Lu
  7. Dahua Lin
  8. Chunyuan Li

In Signal Processing

  1. Yu Qiao
  2. Kaipeng Zhang
  3. Yining Li
  4. Pang Wei Koh
  5. Shiori Sagawa
  6. Ming-Hsuan Yang
  7. Jiaya Jia
  8. Ping Luo

Citations in the corpus