Visual Instruction Tuning

By Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee. Neural Information Processing Systems, 2023

10,396 citations recorded by the source corpus.

Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

View Paper: https://arxiv.org/abs/2304.08485

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Percy Liang
  4. Sergey Levine
  5. Yoshua Bengio
  6. Quoc V. Le
  7. Luke S. Zettlemoyer
  8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Alan L. Yuille

In Multimodal Learning

  1. Jianfeng Gao
  2. Zhe Gan
  3. Hongsheng Li
  4. Trevor Darrell
  5. Devi Parikh
  6. Yu Qiao
  7. Chunyuan Li
  8. Jianwei Yang

Citations in the corpus