BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
By Junnan Li, Dongxu Li, Silvio Savarese, Steven C. H. Hoi. 2023
8,665 citations recorded by the source corpus.
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Sergey Levine
- Yoshua Bengio
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yu Qiao
In Large Language Models
In Multimodal Learning
Citations in the corpus
- Visual Instruction Tuning (2023, methodology): “More sophisticated (but expensive) schemes to connect the image and language representations can also be considered, such as gated cross-attention in Flamingo [2] and Q-former in BLIP-2 [25], or other vision encoders such as SAM [21] that provide object-level features.”
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “Given the significant differences in the model architecture and training data between them, the root cause of the disparity in their capabilities remains elusive, despite conjectures [36, 54]: the amount of training data, the usage of resamplers like Qformer [31], etc .”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Both BLIP [32] and BLIP-2 [31] adopt a costly pre-training stage on additional datasets for superior performance, including Visual Genome [29], Conceptual Captions [58, 7] and LAION [57].”
- ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “Due to the practical difficulty of using these methods, the field has primarily moved towards end-to-end all-in-one models [1, 22, 23, 30].”
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “For instance, BLIP Model [22] is an expert in understanding and providing the description of an image.”
- Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “More importantly, unlocking multimodal input [2, 3, 4, 5, 6, 7] greatly widens the applications of language models to more high-value areas, such as multimodal machine learning, document intelligence, and robotics.”
- MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, methodology): “Existing vision-centric MLLMs follow the paradigm that utilizing pre-trained LLMs and visual encoder with additional learnable modules (Q-former [21, 34, 36, 82] or simple projection layer [22, 39, 43, 61]).”
- mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “• To reduce the number of tokens, Mini-GPT4 (Zhu et al., 2023), mPLUG-Owl (Ye et al., 2023b), and Qwen-VL (Bai et al., 2023) adopt a structure similar to Q-Former (Li et al., 2023a), compressing the token count to a fixed size through learnable queries and cross-attention with visual features.”
- VideoChat: chat-centric video understanding (2023, neutral): “VideoChat-Embed is an end-to-end model designed to handle video-based dialogue.”
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “For instance, models like BLIP [14, 29] and LLaVA [33] require 32 and over 256 tokens respectively for a single image.”
- ChatGPT for Robotics: Design Principles and Model Abilities (2023, methodology): “Recent large models such as BLIP-2 [21], LLaVa [22] have shown that fusing visual and language features into the same representation allows for capabilities such as visual question answering and captioning.”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “In contrast, MiniGPT-4 freezes both the vision encoder (BLIP-2’s ViT + Q-Former [7]) and the Vicuna LLM [108], only training the linear layer.”
- MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023, background): “These approaches [2, 6, 25, 30, 3, 37, 7, 54, 53, 28] connect models from different modalities into integrated end-to-end trainable models, also known as multi-modal foundation models.”
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, methodology): “In recent months, Multimodal Large Language Models (MLLMs) have witness remarkable progress (Alayrac et al., 2022; Huang et al., 2023; Liu et al., 2023a; Zhu et al., 2023; Li et al., 2023a; Gao et al., 2023; Dai et al., 2023).”
- Improving CLIP Training with Language Rewrites (2023, background): “BLIP series [33, 32] include additional captioners and incorporates iterative image captioning within the training pipeline, which intricately link the generated captions with the image content.”
- A Comprehensive Overview of Large Language Models (2023, neutral): “A model trained on limited sequence lengths fails to generalize to unseen lengths at inference time [167], [168].”
- LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, critical): “Third , we find that Otter-Image, BLIVA, and BLIP2 cannot follow the user’s instructions well and generate wrong reasoning steps sometimes, They attain a much worse ranking in the arena than quantitative evaluation because responses with hallucination would have a negative influence on user experience”
- TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, positive): “TinyVLA encompasses several crucial designs: 1) We adopt a pre-trained VLM as the initialization of a policy network; 2) During training the robot data, we freeze the pre-trained parts and utilize the parameter-efficient fine-tuning technique LoRA [29], where the trainable parameters account for only…”