Open Vocabulary Object Detection with Pseudo Bounding-Box Labels

By Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, Caiming Xiong. 2021

118 citations recorded by the source corpus.

Despite great progress in object detection, most existing methods work only on a limited set of object categories, due to the tremendous human effort needed for bounding-box annotations of training data. To alleviate the problem, recent open vocabulary and zero-shot detection methods attempt to detect novel object categories beyond those seen during training. They achieve this goal by training on a pre-defined base categories to induce generalization to novel objects. However, their potential is still constrained by the small set of base categories available for training. To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs. Our method leverages the localization ability of pre-trained vision-language models to generate pseudo bounding-box labels and then directly uses them for training object detectors. Experimental results show that our method outperforms the state-of-the-art open vocabulary detector by 8% AP on COCO novel categories, by 6.3% AP on PASCAL VOC, by 2.3% AP on Objects365 and by 2.8% AP on LVIS. Code is available at https://github.com/salesforce/PB-OVD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Jianwei Yang
3. Lei Zhang
4. Feng Li
5. Hao Zhang
6. Xueyan Zou
7. Zhe Gan
8. Devi Parikh

In Computer Vision and Pattern Recognition

1. Lei Zhang
2. Jianfeng Gao
3. Jianwei Yang
4. Luc Van Gool
5. Dacheng Tao
6. Ming-Hsuan Yang
7. Chunhua Shen
8. Trevor Darrell

In Multimodal Learning

1. Jianfeng Gao
2. Jianwei Yang
3. Lei Zhang
4. Feng Li
5. Hao Zhang
6. Xueyan Zou
7. Sheng Jin
8. Siyi Liu

Citations in the corpus

• A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “…of strong zero-shot transferability demonstrated in the seminal work CLIP [40], a number of researchers have attempted to build advanced open-vocabulary models by leveraging large-scale image-text pairs for fine-grained vision tasks such as detection [10,12,22,30,57] and segmentation [7,17,48,59].”
• ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding (2022, unknown): “We follow prior works [10, 13] that use 63 prompts such as ”a picture of [WORD]” in image-text pre-training tasks and additionally add a dedicated prompt “a point cloud model of [WORD]” to accommodate the 3D modality.”
• Open-vocabulary Attribute Detection (2022, methodology): “Various followup works [7, 17, 18, 38, 57] have improved the performance of open-vocabulary object detection.”
• Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “For example, PB-OVD [189] trains object detectors with VLM-predicted pseudo bounding boxes while XPM [194] introduces a robust cross-modal pseudo-labeling strategy that employs VLM-generated pseudo masks for open-vocabulary instance segmentation.”