DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection

By Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, W. Zhang, Zhenguo Li, Chunjing Xu, Hang Xu. 2022

265 citations recorded by the source corpus.

Open-world object detection, as a more general and challenging goal, aims to recognize and localize objects described by arbitrary category names. The recent work GLIP formulates this problem as a grounding problem by concatenating all category names of detection datasets into sentences, which leads to inefficient interaction between category names. This paper presents DetCLIP, a paralleled visual-concept pre-training method for open-world detection by resorting to knowledge enrichment from a designed concept dictionary. To achieve better learning efficiency, we propose a novel paralleled concept formulation that extracts concepts separately to better utilize heterogeneous datasets (i.e., detection, grounding, and image-text pairs) for training. We further design a concept dictionary~(with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept. By enriching the concepts with their descriptions, we explicitly build the relationships among various concepts to facilitate the open-domain learning. The proposed concept dictionary is further used to provide sufficient negative concepts for the construction of the word-region alignment loss\, and to complete labels for objects with missing descriptions in captions of image-text pair data. The proposed framework demonstrates strong zero-shot detection performances, e.g., on the LVIS dataset, our DetCLIP-T outperforms GLIP-T by 9.9% mAP and obtains a 13.5% improvement on rare categories compared to the fully-supervised model with the same backbone as ours.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Xiaodan Liang
3. Jianwei Yang
4. Lei Zhang
5. Xinggang Wang
6. Zhenguo Li
7. Feng Li
8. Hao Zhang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Lei Zhang
3. Jianfeng Gao
4. Dacheng Tao
5. Xiaodan Liang
6. Ming-Hsuan Yang
7. Jianwei Yang
8. Chunhua Shen

In Multimodal Learning

1. Jianfeng Gao
2. Jianwei Yang
3. Xiaodan Liang
4. Lei Zhang
5. Ying Shan
6. Yixiao Ge
7. Feng Li
8. Hao Zhang

Citations in the corpus

• A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “RegionCLIP [56] and DetCLIP [50] generate pseudo box labels from image-text pairs for more generalized detection.”
• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “Grounding [62] FasterRCNN[39] - sentence partiallabel partiallabel FindIt[21] FasterRCNN[39] A sentence partiallabel fine-tune MDETR[18] DETR[2] A,C word fine-tune zero-shot fine-tune DQ-DETR[46] DETR[2] A,C word zero-shot fine-tune GLIP[26] DyHead[5] A word zero-shot zero-shot zero-shot GLIPv2[59] DyHe”
• Scaling Open-Vocabulary Object Detection (2023, methodology): “Prior work using image captions as weak supervision for detection often used grammatical parsing to extract noun phrases or concepts [40, 33, 32].”
• DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment (2023, methodology): “Similar to DetCLIP [48], DetCLIPv2 is built using the vanilla ATSS [53] detector equipped with a transformer-based [33, 34] text encoder.”
• Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “Alignment objectives align the image-text pair via global image-text matching [71], [72] or local region-word matching [45], [67] on embedding space.”
• YOLO-World: Real-Time Open-Vocabulary Object Detection (2024, methodology): “Several methods [24, 30, 56, 57, 60] reformulate object detection training as region-level vision-language pre-training and train open-vocabulary object detectors at scale.”