BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

By Junnan Li, Dongxu Li, Silvio Savarese, Steven C. H. Hoi. 2023

8,665 citations recorded by the source corpus.

The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Sergey Levine
  4. Yoshua Bengio
  5. Percy Liang
  6. Quoc V. Le
  7. Luke S. Zettlemoyer
  8. Trevor Darrell

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Trevor Darrell
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Yu Qiao

In Large Language Models

  1. Furu Wei
  2. Li Dong
  3. Mohit Bansal
  4. Saining Xie
  5. Yu Qiao
  6. Ping Luo
  7. Hongsheng Li
  8. Shaohan Huang

In Multimodal Learning

  1. Zhe Gan
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Lijuan Wang
  5. Yu Qiao
  6. Cordelia Schmid
  7. Linjie Li
  8. Devi Parikh

Citations in the corpus