Grounded Language-Image Pre-training

By Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang and 2 more. 2021

1,697 citations recorded by the source corpus.

This paper presents a grounded language-image pretraining (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object detection and phrase grounding for pre-training. The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representations semantic-rich. In our experiments, we pre-train GLIP on 27M grounding data, including 3M human-annotated and 24M web-crawled image-text pairs. The learned representations demonstrate strong zero-shot and few-shot transferability to various object-level recognition tasks. 1) When directly evaluated on COCO and LVIS (without seeing any images in COCO during pre-training), GLIP achieves 49.8 AP and 26.9 AP, respectively, surpassing many supervised baselines.11Supervised baselines on COCO object detection: Faster-RCNN w/ ResNet50 (40.2) or ResNet101 (42.0), and DyHead w/ Swin-Tiny (49.7). 2) After fine-tuned on COCO, GLIP achieves 60.8 AP on val and 61.5 AP on test-dev, surpassing prior SoTA. 3) When transferred to 13 downstream object detection tasks, a 1-shot GLIP rivals with a fully-supervised Dynamic Head. Code will be released at https://github.com/microsoft/GLIP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Visual Instruction Tuning (2023, background): “…foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44, 30].”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Large-scale Language Models (LLMs) [13, 52, 73, 53, 15] have stimulated widespread attention in both academia and industry.”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “…large amount of image-text data for open-vocabulary detection or segmentation, such as distilling the visual-semantic representations from multi-modal foundation models [12, 57], designing fine-grained or augmented contrastive learning methods [39] or utilizing pseudo-labeling techniques [30, 56].”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): ““car” or “golden retriever”), and returns a list of image patches containing the noun phrase. exists ( GLIP [31] ) takes as input an image and a short noun phrase and returns a boolean indicating whether an instance of that noun phrase is present in the image.”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, methodology): “Inspired by GLIP [26], we add an image-to-text cross-attention and a text-to-image cross-attention for feature fusion.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “Note that GLIPv2 uses over 10M pretraining data, including around 2M with box supervision.”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, methodology): “In GLIP, its linear probing has been implemented via updating Wv and Wt in [34].”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “Many recent works aim to transfer the language capabilities of these models to object detection [12,26,45,46,20].”
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models (2022, background): “The idea of TPT can be applied to other foundation models for various downstream tasks, including other vision-language models [6, 74] and foundation models of other modalities ( e.g. , pre-trained large-scale language models [19, 18]) to further boost their zero-shot generalization.”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “Recently, many large companies and research institutions including OpenMind [41], Microsoft [42], [43]”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “8(b) shows an overview”
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, neutral): “Since then there has been a surge towards building a new category of classiﬁers, known as zero-shot or open-vocabulary clas-siﬁers, that are capable of detecting a wide range of class categories [25, 47, 48, 1].”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “At this stage, the model takes high-resolution (800 × 1,333) images as input, and is pre-trained with bounding box localization loss and word-region alignment loss, as used in GLIP [34].”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, neutral): “One approach focuses on leveraging existing pretrained task-specific tools such as image captioners [15], object detectors [59] and text-to-image generators [60] by either creating multimodal prompt interfaces [43, 61] or by executing LLM-generated programs [62, 63, 64].”
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, methodology): “Specifically, we first use the object detector GLIP [12] to locate the objects, and then use the cropped region proposals to retrieve different types of external knowledge.”