SVIT: Scaling up Visual Instruction Tuning
By Bo Zhao, Boya Wu, Tiejun Huang. 2023
176 citations recorded by the source corpus.
Thanks to the emerging of foundation models, the large language and vision models are integrated to acquire the multimodal ability of visual captioning, question answering, etc. Although existing multimodal models present impressive performance of visual understanding and reasoning, their limits are still largely under-explored due to the scarcity of high-quality instruction tuning data. To push the limits of multimodal capability, we Scale up Visual Instruction Tuning (SVIT) by constructing a dataset of 4.2 million visual instruction tuning data including 1.6M conversation question-answer (QA) pairs, 1.6M complex reasoning QA pairs, 1.0M referring QA pairs and 106K detailed image descriptions. Besides the volume, the proposed dataset is also featured by the high quality and rich diversity, which is generated by prompting GPT-4 with the abundant manual annotations of images. We also propose a new data recipe to select subset with better diversity and balance, which evokes model's superior capabilities. Extensive experiments verify that SVIT-v1.5, trained on the proposed dataset, outperforms state-of-the-art Multimodal Large Language Models on popular benchmarks. The data and code are publicly available at https://github.com/BAAI-DCAI/Visual-Instruction-Tuning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Multimodal Learning
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “Recent works further demonstrate improved performance by scaling up the pretraining data [3, 13, 53], instruction-following data [13, 17, 28, 57], visual encoders [3], or language models [38], respectively.”
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, unknown): “…ST-VQA (en) [11] Document DocVQA (en) [22], Common Crawl PDF (en&zh) Grounding RefCOCO/+/g (en) [75, 126], Visual Genome (en) [42] LLaVA-150K (en&zh) [59], LVIS-Instruct4V (en) [110], ALLaVA (en&zh) [16], Laion-GPT4V (en) [1], Conversation TextOCR-GPT4V (en) [2], SVIT (en&zh) [135] OpenHermes2.”
- Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…All-Seeing-V2 [250], V3Det [243], TolokaVQA [236] Document DocReason25K [93], DocVQA [184], Docmatix [121], Synthetic Arxiv QA ALLaVA [25], SVIT [309], Cambrain-GPT4o [234], TextOCR-GPT4V [102], MMDU [159], Conversation Synthetic Real-World Conversations PMC-VQA [303], VQA-RAD [120], ImageCLEF…”
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, methodology): “Models such as LLaMA-Adapter [20, 81], mPlug-OWL [74, 75], SVIT [82], LRV-Instruction [39], and InstructBLIP [16] exemplify these developments.”
- mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2023, unknown): “Instruction Tuning with MLLMs.”
- Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, unknown): “Concurrently, to maintain OpenFlamingo’s in-context learning capacity, we retrieve in-context examples for each triplet, which often lack correlated context, such as a visual question-answer data sample in VQAv2 [3].”