SVIT: Scaling up Visual Instruction Tuning

By Bo Zhao, Boya Wu, Tiejun Huang. 2023

176 citations recorded by the source corpus.

Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yu Su
  2. Haotian Liu
  3. Yong Jae Lee
  4. Chunyuan Li
  5. Huan Sun
  6. Boyuan Zheng
  7. Yuheng Li
  8. Kai Zhang

In Computer Vision and Pattern Recognition

  1. Chunyuan Li
  2. Yong Jae Lee
  3. Haotian Liu
  4. Yuheng Li
  5. Yu Su
  6. Huan Sun
  7. Luc Van Gool
  8. Boyuan Zheng

In Multimodal Learning

  1. Chunyuan Li
  2. Haotian Liu
  3. Yong Jae Lee
  4. Yu Su
  5. Huan Sun
  6. Boyuan Zheng
  7. Yuheng Li
  8. Saining Xie

Citations in the corpus