Improved Baselines with Visual Instruction Tuning
By Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee. 2023
5,499 citations recorded by the source corpus.
Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this paper, we present the first systematic study to investigate the design choices of LMMs in a controlled setting under the LLaVA framework. We show that the fully-connected vision-language connector in LLaVA is surprisingly power-ful and data-efficient. With simple modifications to LLa VA, namely, using CLIP- ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~ 1 day on a single 8-AI00 node. Furthermore, we present some early exploration of open problems in LMMs, including scaling to higher resolution inputs, compositional capabilities, and model hallucination, etc. We hope this makes state-of-the-art LMM research more accessible. Code and model will be publicly available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Quoc V. Le
- Percy Liang
- Yoshua Bengio
- Sergey Levine
- Luke S. Zettlemoyer
- Graham Neubig
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Alan L. Yuille
- Bernt Schiele
In Large Language Models
- Noah A. Smith
- Graham Neubig
- Luke S. Zettlemoyer
- Maosong Sun
- Percy Liang
- Zhiyuan Liu
- Huan Sun
- Hongsheng Li
In Multimodal Learning
Citations in the corpus
- Visual Instruction Tuning (2023, neutral): “For more quantitative results of LLaVA on academic benchmarks, please refer to the improved baselines with visual instruction tuning [32].”
- Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “5 [45] places image tokens (MLP-projected CLIP visual tokens) ahead of question tokens.”
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, background): “5 [32], Gemini Pro Vision [39], Claude 3 [4], MiniGPT4 [63], MiniGPT5 [62], GiT [44], InstructBLIP [12], Qwen-VL [7], mPLUG-Owl-v1 [50], mPLUG-Owl-v2 [51], LRV-Instruction [29], BLIP2 [22], BLIP2-T5 [22], and Open-Flamingo [3].”
- SVIT: Scaling up Visual Instruction Tuning (2023, neutral): “5-mix-665K (Liu et al., 2023a) with SVIT-core-150K.”
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “…understanding component, a stronger vision encoder can be chosen without worrying about whether the encoder is capable of handling vision generation tasks, such as EVA-CLIP [71], InternViT [13], etc. (2) To handle high-resolution images, dynamic high-resolution techniques [48] can be used.”
- ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, methodology): “5 [35], which uses CLIP-Large [50] as the visual encoder, a two-layer MLP as the connector, and Vicuna-v1.”
- On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “…low cost, this generation paradigm has also exhibited significant promise in multimodal tasks, including text-image retrieval (Kritharoula et al., 2023), chat understanding (Han et al., 2023), visual question answering (Han and Gardent, 2023), and multimodal instruction tuning (Liu et al., 2023).”
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, methodology): “With the increasing scale of large language model backends, approaches such as InstructBLIP [9] and LLaVA [29, 30] collecte the larger human instruction datasets to train a larger LVLMs ( e.g .”
- SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “To enhance this approach, we prepend the object class to each question.”
- VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, neutral): “5-7B (Liu et al., 2023a).”
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, neutral): “5 [30] and Shikra [5].”
- MMBench: Is Your Multi-modal Model an All-around Player? (2023, methodology): “…on MMBench: (a) Text-Only GPT-4 [37]; (b) Open-Source VLMs including model variants of OpenFlamingo [4], MiniGPT4 [56], InstructBLIP [12], LLaVA [32], IDEFICS [26], CogVLM [47], Qwen-VL [6], Yi-VL [2], mPLUG-Owl [50], InternLM-XComposer [13], and MiniCPM-V [39]; (c) Proprietary VLMs including…”
- CogAgent: A Visual Language Model for GUI Agents (2023, neutral): “5 [20] Vicuna-13B 36.3 84.5 Emu [36] LLaMA-13B hallucinations compared to other models.”
- Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception (2024, background): “As Multimodal Large Language Models (MLLM) Liu et al. [2023d], Zhu et al. [2023], Ye et al. [2023a], Dai et al. [2023], Liu et al. [2023e], Chen et al. [2023], Ye et al. [2023b], Bai et al. [2023], Lin et al. [2023] rapidly progress and exhibit remarkably visual comprehension capabilities, the…”
- Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “Built upon the CLIP image encoder which is somewhat aligned with the language modality, current LVLMs typically utilize vast image-text pairs to connect the vision encoder and LLM, enabling LLM to receive and understand visual content [54, 26, 24, 11, 52, 2, 48, 31, 5].”