Otter: A Multi-Modal Model With In-Context Instruction Tuning
By Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, Ziwei Liu. 2023
692 citations recorded by the source corpus.
Recent advances in Large Multimodal Models (LMMs) have unveiled great potential as visual assistants. However, most existing works focus on responding to individual instructions or using previous dialogues for contextual understanding. There is little discussion on employing both images and text as in-context examples to enhance the instruction following capability. To bridge this gap, we introduce the Otter model to leverage both textual and visual in-context examples for instruction tuning. Specifically, Otter builds upon Flamingo with Perceiver architecture, and has been instruction tuned for general purpose multi-modal assistant. Otter seamlessly processes multi-modal inputs, supporting modalities including text, multiple images, and dynamic video content. To support the training of Otter, we present the MIMIC-IT (MultI-Modal In-Context Instruction Tuning) dataset, which encompasses over 3 million multi-modal instruction-response pairs, including approximately 2.2 million unique instructions across a broad spectrum of images and videos. MIMIC-IT has been carefully curated to feature a diverse array of in-context examples for each entry. Comprehensive evaluations suggest that instruction tuning with these in-context examples substantially enhances model convergence and generalization capabilities. Notably, the extensive scenario coverage provided by the MIMIC-IT dataset empowers the Otter model to excel in tasks involving complex video and multi-image understanding.
Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF, Multimodal Learning, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Large Language Models
In Alignment & RLHF
In Multimodal Learning
In Signal Processing
Citations in the corpus
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The current working directory is /home/user/ echo “I am a chatbot“ > file.txt && cat file.txt && ls -l I am a chatbot -rw-r–r– 1 user user”
- Aligning Large Multimodal Models with Factually Augmented RLHF (2023, methodology): “A.2 Data Filtering v.s. RLHF In our preliminary tests, we employed the Fact-RLHF reward model to filter out 70%, 50%, and 30% of LLaVA data.”
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (2023, background): “…et al., 2023c; Dai et al., 2023; Huang et al., 2023; Peng et al., 2023; Zhu et al., 2023; Liu et al., 2023; Ye et al., 2023b,a; Chen et al., 2023a; Li et al., 2023a; Zhang et al., 2023; Sun et al., 2023; OpenAI, 2023) have been developed to enhance large language models with the ability to…”
- SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension (2023, background): “Leveraging the strong generality of LLMs, generative Multimodal Large Language Models (MLLMs) [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21] have demonstrate enhanced abilities for multimodal comprehension and generation.”
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “The development of open-source MLLMs [4, 15, 20, 43, 47, 49, 52, 65, 66, 98, 105, 113, 115, 119, 133, 134] has significantly influenced the AGI landscape by integrating and enhancing capabilities in processing both visual and textual data.”
- An Embodied Generalist Agent in 3D World (2023, neutral): “Furthermore, results on HM3D-val confirm LEO’s zero-shot generalization to novel scenes.”
- A survey on multimodal large language models (2023, neutral): “5) Learnable Interface Query Based Video-LLaMA [42], MultiModal-GPT [31], Macaw-LLM [41], X-LLM [25], ChatBridge [29], mPLUG-Owl [22], Otter [27], VideoChat-Embed [33], MiniGPT-4 [13], Ying-VLM [36], InstructBLIP [23], VisionLLM [24], MedVInT-TD [30]”
- MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “This category includes: Otter-V [18], mPlug-Owl-V [48], Video LLaMA-2 [52], Video ChatGPT [28], VideoChat [20], VideoChat2 [21], and Video-LLaVA [23].”
- DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “• In addition, since the diffusion model is used as the image decoder, it would be interesting to explore DMs-based zero-shot perception such as classification (Li et al., 2023a) and segmentation (Xu et al., 2023a) with the goal of unifying all perception tasks with image synthesis/generation.”
- VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, methodology): “The other approach [13, 16, 67, 45, 68, 17, 11] focuses on building a single pretrained model that can follow instructions by supervised finetuning on multimodal vision-language data.”
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, neutral): “We then dive deeper into the open-sourced LMMs and examine how the training data, vision encoder, and LLM selection influence the performance on different capabilities.”
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “Open-Flamingo [3], derived from Flamingo [1], and Otter [30], an in-context instruction tuning variant of OpenFlamingo, are also included in our comparison.”
- RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, background): “Recent trends in multimodal learning have witnessed the success of building MLLMs by connecting visual encoders with powerful LLMs [8, 16, 18, 45, 49].”