BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
By Junnan Li, Dongxu Li, Caiming Xiong, Steven C. H. Hoi. 2022
7,032 citations recorded by the source corpus.
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Sergey Levine
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Trevor Darrell
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Large Language Models
In Multimodal Learning
Citations in the corpus
- The False Promise of Imitating Proprietary LLMs (2023, methodology): “For example, the training process of many recent vision-language model (Li et al., 2022; Liu et al., 2023; Ye et al., 2023; Zhu et al., 2023) includes ChatGPT or GPT-4 outputs at some stages.”
- Survey of Hallucination in Natural Language Generation (2022, background): “With the vast advancement of the Transformer architecture [35, 189] in both CV and NLP, there is a trend to pre-train large-scale unified vision-language (VL) models [1, 27, 105, 196, 197, 200] to perform vision grounded text generation tasks, such as image captioning and visual question answering (”
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, positive): “Note that instead of video-to-text retrieval, but rather on textto-video retrieval, a recent method (Li et al. 2022) has shown strong zero-shot results.”
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Concurrent works [17, 58, 119, 124, 154] also propose to formulate numerous vision tasks as text generation problems.”
- Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, background): “There are also VLP models that can perform multimodal generation tasks, by expensive pretraining with objective of image-conditioned autoregressive language modeling (Lin et al., 2021; Wang et al., 2021; Hu et al., 2021; Li et al., 2022).”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Both BLIP [32] and BLIP-2 [31] adopt a costly pre-training stage on additional datasets for superior performance, including Visual Genome [29], Conceptual Captions [58, 7] and LAION [57].”
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “Remove Objects from Image [10, 35] Replace Objects from Image [10, 35] Change Image by the Text [35] Image Question Answering [23] Image-to-Text [23] Text-to-Image [35] Image-to-Edge [45] Edge-to-Image [53] Image-to-Line [16] Line-to-Image [53] Image-to-Hed [44] Hed-to-Image [53] Image-to-Seg [24] S”
- Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “We randomly picked up 50 samples whose prediction results were correct and 50 samples whose prediction results were incorrect.”
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, positive): “Inspired by BLIP (Li et al., 2022), we jointly optimize three pre-training objectives that share the same input format and model parameters.”
- GIT: A Generative Image-to-text Transformer for Vision and Language (2022, neutral): “Compared with the existing approaches (Wang et al., 2021a;b; Zhang et al., 2021a; Li et al., 2022b) for VQAv2 (Goyal et al., 2017), our model is generative without pre-defining the candidate answers, even in inference.”
- Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, methodology): “After pretraining, several models exhibit zero-shot capabilities in image-text retrieval (Jia et al., 2021; Radford et al., 2021; Zeng et al., 2022b) and image captioning (Wang et al., 2022; Li et al., 2022b).”
- MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “…pushing the performance boundaries across a spectrum of downstream tasks (Li et al., 2020; Akbari et al., 2021; Fang et al., 2021; Yan et al., 2021; Li et al., 2021; Radford et al., 2021; Li et al., 2022; Zellers et al., 2022; Zeng et al., 2022b; Yang et al., 2022; Wang et al., 2022a,b).”
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (2023, background): “In recent years, the trend of using autoregressive language models as decoders in vision-language tasks has gained significant traction [6, 15, 36, 31, 2, 16, 17, 12].”
- Zero-shot Image-to-Image Translation (2023, methodology): “To invert a text-conditioned model, we generate an initial text prompt c using BLIP [33] to describe the input image ˜ x .”
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, methodology): “3, for the stage of understanding pretraining, we use a subset of 558K LAION-CC-SBU image-text pairs with BLIP [25] captions, which is sourced from CC3M [39] and filtered by Liu et al .”
- Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, background): “…provides a plausible explanation for our impressive evaluation results on conventional benchmarks and a comprehensive benchmark for MLLMs. Previous research (Li et al., 2022a; Lee et al., 2021; Bugliarello et al., 2023a) has underscored the significance of data curation in bolstering performance.”