InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

By Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, Steven C. H. Hoi. 2023

3,705 citations recorded by the source corpus.

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Furu Wei
  4. Zhe Gan
  5. Percy Liang
  6. Luke S. Zettlemoyer
  7. Yoshua Bengio
  8. Quoc V. Le

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Ming-Hsuan Yang
  5. Trevor Darrell
  6. Yu Qiao
  7. Jitendra Malik
  8. Andrew Zisserman

In Alignment & RLHF

  1. Noah A. Smith
  2. Percy Liang
  3. Yushi Hu
  4. Hannaneh Hajishirzi

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Linjie Li
  4. Zicheng Liu
  5. Jianfeng Wang
  6. Jianfeng Gao
  7. Chunyuan Li
  8. Zhengyuan Yang

Citations in the corpus