InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
By Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, Steven C. H. Hoi. 2023
3,705 citations recorded by the source corpus.
Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Alignment & RLHF, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Yu Qiao
- Jitendra Malik
- Andrew Zisserman
In Alignment & RLHF
In Multimodal Learning
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “Following InstructBLIP [13], the image subset of ScienceQA [37] with multiple choice are used to evaluate the zero-shot generalization on scientific question answering.”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The current working directory is /home/user/ echo “I am a chatbot“ > file.txt && cat file.txt && ls -l I am a chatbot -rw-r–r– 1 user user”
- Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “Meanwhile, some work [57, 58, 59] introduce visual instruction tuning to enhance instruction following capabilities.”
- Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, background): “, 2023; Qwen, 2023), researchers have started building more powerful large vision-language models (LVLMs) based on LLMs (Alayrac et al., 2022; Chen et al., 2022; Li et al., 2023b; Dai et al., 2023; Huang et al., 2023; Peng et al., 2023; Zhu et al., 2023; Liu et al., 2023; Ye et al., 2023b,a; Chen et”
- Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, methodology): “Although it is marginally outperformed by InstructBLIP (Dai et al., 2023), K OSMOS -2 surpasses the other models in the comparison.”
- Evaluating Object Hallucination in Large Vision-Language Models (2023, methodology): “They can be formulated as: We select five recently released LVLMs, i.e., mPLUG-Owl (Ye et al., 2023), LLaVA (Liu et al., 2023), Multimodal-GPT (Gong et al., 2023), MiniGPT-4 (Zhu et al., 2023) and Instruct-BLIP (Dai et al., 2023a) and prompt them with following instructions to generate captions abou”
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (2023, neutral): “…language models specified in parentheses: MiniGPT-4 (Vicuna 13B) (Zhu et al., 2023), LLaVa (LLaMA 13B) (Liu et al., 2023d), MMGPT (LLaMA 7B) (Gong et al., 2023), LLaMA-Adapter (LLaMA 7B) (Zhang et al., 2023b), mPLUG-Owl (LLaMA 7B) (Ye et al., 2023), and InstructBLIP (Vicuna 7B) (Dai et al., 2023).”
- Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception (2024, background): “As Multimodal Large Language Models (MLLM) Liu et al. [2023d], Zhu et al. [2023], Ye et al. [2023a], Dai et al. [2023], Liu et al. [2023e], Chen et al. [2023], Ye et al. [2023b], Bai et al. [2023], Lin et al. [2023] rapidly progress and exhibit remarkably visual comprehension capabilities, the…”
- Qwen2.5-Omni Technical Report (2025, neutral): “…(LALMs) (OpenAI, 2024; Tang et al., 2024; Chu et al., 2023b; 2024b) and Language-Visual-Language Models (LVLMs) (Li et al., 2023; Liu et al., 2023b; Dai et al., 2023; Zhu et al., 2023; Huang et al., 2023; Bai et al., 2023b; Liu et al., 2023a; Wang et al., 2023b; OpenAI., 2023; Gemini Team, 2024)…”
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, background): “This process emulates the ongoing development of visual language models (Zhu et al., 2023; Dai et al., 2023; Liu et al., 2023a) via fine-tuning of off-the-shelf unimodal models.”
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “To effectively aggregate features from multiple vision encoders and prevent the information loss introduced by interpolation, we use a set of learnable latent queries that interact with multiple vision features via cross-attention layers [31].”
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “For Blip2 and Instructblip, We concatenate the visual prompts of all frames and place them on the top of the textual prompts following (Dai et al., 2023).”
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, neutral): “Different phrasings of requests for the same behavior can be highly informative to investigate, but we focus on unique underlying behaviors for evaluation purposes and consider rephrasing of requests to be a potential component of red teaming methods rather than a feature of the evaluation.”
- mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2023, background): “∗ Equal contribution † Corresponding author 1 https://openai.com/research/gpt-4v-system-card benchmarks using a single generic model .”