Open-Vocabulary DETR with Conditional Matching

By Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, Chen Change Loy. 2022

319 citations recorded by the source corpus.

Open-vocabulary object detection, which is concerned with the problem of detecting novel objects guided by natural language, has gained increasing attention from the community. Ideally, we would like to extend an open-vocabulary detector such that it can produce bounding box predictions based on user inputs in form of either natural language or exemplar image. This offers great flexibility and user experience for human-computer interaction. To this end, we propose a novel open-vocabulary detector based on DETR -- hence the name OV-DETR -- which, once trained, can detect any object given its class name or an exemplar image. The biggest challenge of turning DETR into an open-vocabulary detector is that it is impossible to calculate the classification cost matrix of novel classes without access to their labeled images. To overcome this challenge, we formulate the learning objective as a binary matching one between input queries (class name or exemplar image) and the corresponding objects, which learns useful correspondence to generalize to unseen queries during testing. For training, we choose to condition the Transformer decoder on the input embeddings obtained from a pre-trained vision-language model like CLIP, in order to enable matching for both text and image queries. With extensive experiments on LVIS and COCO datasets, we demonstrate that our OV-DETR -- the first end-to-end Transformer-based open-vocabulary detector -- achieves non-trivial improvements over current state of the arts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lei Zhang
2. Xiaodan Liang
3. Ziwei Liu
4. Fahad Shahbaz Khan
5. Jianwei Yang
6. Xinggang Wang
7. Chen Change Loy
8. Jianfeng Gao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Chen Change Loy
4. Lei Zhang
5. Dacheng Tao
6. Xiaogang Wang
7. Alan L. Yuille
8. Chunhua Shen

In Multimodal Learning

1. Xiaodan Liang
2. Jianwei Yang
3. Ying Shan
4. Lei Zhang
5. Yixiao Ge
6. Feng Li
7. Hao Zhang
8. Zhuowen Tu

Citations in the corpus

• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “Grounding [62] FasterRCNN[39] - sentence partiallabel partiallabel FindIt[21] FasterRCNN[39] A sentence partiallabel fine-tune MDETR[18] DETR[2] A,C word fine-tune zero-shot fine-tune DQ-DETR[46] DETR[2] A,C word zero-shot fine-tune GLIP[26] DyHead[5] A word zero-shot zero-shot zero-shot GLIPv2[59] DyHe”
• Open-Vocabulary Panoptic Segmentation with MaskCLIP (2022, methodology): “Many algorithms have been developed lately [34, 31, 36, 24, 26, 28] trying knowledge distillation from the CLIP model to benefit the down-stream tasks one way or the other by leveraging the rich semantic language information paired in the images.”
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, background): “VL models possess a strong understanding of open-vocabulary concepts, making them suitable for various downstream vision and vision-language applications [12, 56, 38, 30, 60, 13, 32, 53, 26, 36, 8].”
• DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection (2022, background): “Intuitively, some works [15, 53, 50, 12] try to extend a two-stage detector to an open-world detector by distilling the learned image embeddings of the cropped proposal regions from a pretrained VL model.”
• Transformer-Based Visual Segmentation: A Survey (2023, background): “• Open Vocabulary Learning: Recent studies [246], [273], [274], [275], [276], [277], [278] focus on the open vocabulary and open world setting, where their goal is to detect and segment novel classes, which are not seen during the training.”
• YOLO-World: Real-Time Open-Vocabulary Object Detection (2024, background): “Early works [9, 14, 48, 53, 58] follow the standard OVD setting [59] by training detectors on the base classes and evaluating the novel (unknown) classes.”
• Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “For VLM knowledge distillation, many studies adopt different task-specific backbones ( e.g. , ViLD adopts Faster R-CNN, OV-DETR uses DETR) which complicates benchmarking greatly.”
• DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment (2023, background): “Inspired by the success of vision-language pre-training, recent works [16, 44, 51, 54] propose to transfer knowledge of a pre-trained VL model (e.”
• Recognize Anything: A Strong Image Tagging Model (2023, unknown): “We adopt Swin-transformer [19] as the image encoder, as it has demonstrated better performance than naive ViT in both vision-language [10] and tagging domains [18].”
• Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, background): “Most prior works on open-vocabulary segmentation either perform object detection with instance segmentation alone [17, 22, 23, 40, 51, 80, 81, 86, 89] or open-vocabulary semantic segmentation alone [22, 36, 76, 88].”