DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment

By Lewei Yao, Jianhua Han, Xiaodan Liang, Danqian Xu, W. Zhang, Zhenguo Li, Hang Xu. 2023

129 citations recorded by the source corpus.

This paper presents DetCLIPv2, an efficient and scalable training framework that incorporates large-scale imagetext pairs to achieve open-vocabulary object detection (OVD). Unlike previous OVD frameworks that typically rely on a pre-trained vision-language model (e.g., CLIP) or exploit image-text pairs via a pseudo labeling process, DetCLIPv2 directly learns the fine-grained word-region alignment from massive image-text pairs in an end-to-end manner. To accomplish this, we employ a maximum word-region similarity between region proposals and textual words to guide the contrastive objective. To enable the model to gain localization capability while learning broad concepts, DetCLIPv2 is trained with a hybrid supervision from detection, grounding and image-text pair data under a unified data formulation. By jointly training with an alternating scheme and adopting low-resolution input for image-text pairs, DetCLIPv2 exploits image-text pair data efficiently and effectively: DetCLIPv2 utilizes 13 × more image-text pairs than DetCLIP with a similar training time and improves performance. With 13M image-text pairs for pre-training, DetCLIPv2 demonstrates superior open-vocabulary detection performance, e.g., DetCLIPv2 with Swin-T backbone achieves 40.4% zero-shot AP on the LVIS benchmark, which outperforms previous works GLIP/GLIPv2/DetCLIP by 14.4/11.4/4.5% AP, respectively, and even beats its fully-supervised counterpart by a large margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinggang Wang
2. Wenyu Liu
3. Ying Shan
4. Yixiao Ge
5. Matthias Minderer
6. Alexey A. Gritsenko
7. Neil Houlsby
8. Lin Song

In Computer Vision and Pattern Recognition

1. Xinggang Wang
2. Ying Shan
3. Wenyu Liu
4. Luc Van Gool
5. Yixiao Ge
6. Neil Houlsby
7. Alexey A. Gritsenko
8. Chen Change Loy

In Multimodal Learning

1. Ying Shan
2. Yixiao Ge
3. Alexey A. Gritsenko
4. Xinggang Wang
5. Neil Houlsby
6. Matthias Minderer

Citations in the corpus

• Scaling Open-Vocabulary Object Detection (2023, methodology): “Prior work using image captions as weak supervision for detection often used grammatical parsing to extract noun phrases or concepts [40, 33, 32].”
• YOLO-World: Real-Time Open-Vocabulary Object Detection (2024, methodology): “Several methods [24, 30, 56, 57, 60] reformulate object detection training as region-level vision-language pre-training and train open-vocabulary object detectors at scale.”