Improved Baselines with Visual Instruction Tuning

By Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee. 2023

5,499 citations recorded by the source corpus.

Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this paper, we present the first systematic study to investigate the design choices of LMMs in a controlled setting under the LLaVA framework. We show that the fully-connected vision-language connector in LLaVA is surprisingly power-ful and data-efficient. With simple modifications to LLa VA, namely, using CLIP- ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~ 1 day on a single 8-AI00 node. Furthermore, we present some early exploration of open problems in LMMs, including scaling to higher resolution inputs, compositional capabilities, and model hallucination, etc. We hope this makes state-of-the-art LMM research more accessible. Code and model will be publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Quoc V. Le
  4. Percy Liang
  5. Yoshua Bengio
  6. Sergey Levine
  7. Luke S. Zettlemoyer
  8. Graham Neubig

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Ming-Hsuan Yang
  5. Trevor Darrell
  6. Jitendra Malik
  7. Alan L. Yuille
  8. Bernt Schiele

In Large Language Models

  1. Noah A. Smith
  2. Graham Neubig
  3. Luke S. Zettlemoyer
  4. Maosong Sun
  5. Percy Liang
  6. Zhiyuan Liu
  7. Huan Sun
  8. Hongsheng Li

In Multimodal Learning

  1. Hongsheng Li
  2. Peng Gao
  3. Yu Qiao
  4. Aniruddha Kembhavi
  5. Renrui Zhang
  6. Jingren Zhou
  7. Pan Lu
  8. Jifeng Dai

Citations in the corpus