GLIPv2: Unifying Localization and Vision-Language Understanding

By Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, Jianfeng Gao. 2022

392 citations recorded by the source corpus.

We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Experimental results show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. The model also shows (1) strong zero-shot and few-shot adaption performance on open-vocabulary object detection tasks and (2) superior grounding capability on VL understanding tasks. Code will be released at https://github.com/microsoft/GLIP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “…large amount of image-text data for open-vocabulary detection or segmentation, such as distilling the visual-semantic representations from multi-modal foundation models [12, 57], designing fine-grained or augmented contrastive learning methods [39] or utilizing pseudo-labeling techniques [30, 56].”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, methodology): “Grounding DINO outperforms GLIP under the same back-bone for the zero-shot setting, comparable with GLIPv2 [59] without any new techniques like masked training.”
CogVLM: Visual Expert for Pretrained Language Models (2023, positive): “, which extracts nouns via spaCy [Honnibal and Johnson, 2015] and predicts the bounding boxes using GLIPv2 [Zhang et al., 2022].”
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, background): “Information mined from image-text pairs can effectively provide labels for basic visual tasks [34, 39, 74].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “85 GLIPv2 [194] arXiv-2022 image-text Swin, BERT PGL, CS, MLM Serves both the localization and understanding tasks.”
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation (2022, neutral): “Method IN-1K (%) COCO (APbox) COCO (APmask) ADE20K (mIoU) GLIPv2-CoSwin-H [52] 62.”
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection (2022, neutral): “5 GLIPV2-T[54] SWIN-T+DH+F O365,GOLDG,CAP4M 29.”
Scaling Open-Vocabulary Object Detection (2023, methodology): “0 – – – – – – – 5 GLIPv2 [39] Swin-B 48.”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, methodology): “Consequently, inspired by prior successful works such as Grounded Pre-training [81, 38] and SAM [25], we aim to address complex segmentation in the wild tasks by combining the strong open-set foundation models.”
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (2023, methodology): “A series of works [26, 27] were improved upon CLIP by employing refined data strategies for more diverse data, they have been effective for basic visual tasks [28, 32, 59] but less so for complex tasks like visual question answering.”
DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment (2023, methodology): “shot fine-tuning on LVIS [17] with 1203 categories and ODinW13 [27, 52] containing 13 detection tasks.”
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks (2023, methodology): “Florence [70] uses FLOD-9M and GLIP [33,73] is trained with FourODs.”
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, positive): “Leveraging the GLIPv2 [69] model, we parse the caption of each image and automatically predict the corresponding bounding boxes for every noun phrase.”