Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model

By Yu Du, Fangyun Wei, Miaojing Shi, Yue Gao, Guoqi Li. Computer Vision and Pattern Recognition, 2022

465 citations recorded by the source corpus.

Recently, vision-language pre-training shows great potential in open-vocabulary object detection, where detectors trained on base classes are devised for detecting new classes. The class text embedding is firstly generated by feeding prompts to the text encoder of a pre-trained vision-language model. It is then used as the region classifier to supervise the training of a detector. The key element that leads to the success of this model is the proper prompt, which requires careful words tuning and ingenious design. To avoid laborious prompt engineering, there are some prompt representation learning methods being proposed for the image classification task, which however can only be sub-optimal solutions when applied to the detection task. In this paper, we introduce a novel method, detection prompt (DetPro), to learn continuous prompt representations for open-vocabulary object detection based on the pre-trained vision-language model. Different from the previous classification-oriented methods, DetPro has two highlights: 1) a background interpretation scheme to include the proposals in image background into the prompt training; 2) a context grading scheme to separate proposals in image foreground for tailored prompt training. We assemble DetPro with ViLD, a recent state-of-the-art openworld object detector, and conduct experiments on the LVIS as well as transfer learning on the Pascal VOC, COCO, Objects365 datasets. Experimental results show that our DetPro outperforms the baseline ViLD [7] in all settings, e.g., +3.4 APbox and +3.0 APmask improvements on the novel classes of LVIS. Code and models are available at https://github.com/dyabel/detpro.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d6c73f758b05f38529c1a96cab7e908a2047dabd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Open-Vocabulary DETR with Conditional Matching (2022, methodology): “We can follow previous works [13,35,7] to generate class-agnostic object proposals that may cover the novel classes, but we do not know the ground-truth classification labels of these proposals.”
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, unknown): “, 2022) proposes to distill the rich representation of pretrained VLM into the detector, and DetPro (Du et al., 2022) improves upon ViLD by applying the idea of prompt optimization.”
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models (2022, methodology): “Although initially developed for language models, prompt tuning has later been applied to other domains, including vision-language models [28, 15, 29] and continual learning [30].”
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, methodology): “The textual prompt tuning is also widely used in open-vocabulary object detection [14] and semantic segmentation [40].”
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection (2022, background): “Intuitively, some works [15, 53, 50, 12] try to extend a two-stage detector to an open-world detector by distilling the learned image embeddings of the cropped proposal regions from a pretrained VL model.”
Unsupervised Prompt Learning for Vision-Language Models (2022, methodology): “…al. 2021) and CoOp (Zhou et al. 2021), we use 11 publicly available image classification datasets including ImageNet (Deng et al. 2009), Caltech101 (Fei-Fei, Fergus, and Perona 2004), DTD (Cimpoi et al. 2014), EuroSAT (Helber et al. 2019), FGVCAircraft (Maji et al. 2013), Food101 (Bossard,…”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “Although primarily focusing on processing and understanding textual information, some LLM models also have remarkable image processing capabilities in vision-related tasks [136], including image-to-text generation [137] and object detection [138], etc.”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “For example, DetPro [37] introduces a detection prompt technique for learning continuous prompt representations for open-vocabulary object detection.”
YOLO-World: Real-Time Open-Vocabulary Object Detection (2024, methodology): “For the LVIS dataset, we follow previous works [9, 14, 64] and fine-tune YOLO-World on the LVIS-base (common & frequent) and evaluate it on the LVIS-novel (rare).”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, background): “Most prior openvocabulary recognition works either perform object detection with instance segmentation alone [18, 23, 25, 44, 55, 87, 88, 93, 96] or open-vocabulary semantic segmentation alone [23, 40, 83, 95].”