Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

By Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, Yin Cui. 2021

1,353 citations recorded by the source corpus.

We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. It is costly to further scale up the number of classes contained in existing object detection datasets. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories that are not seen during training. ViLD obtains 16.1 mask AP$_r$ with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with a stronger teacher model ALIGN, ViLD achieves 26.3 AP$_r$. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP$_{50}$ on PASCAL VOC, 36.6 AP on COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous state-of-the-art by 4.8 on novel AP and 11.4 on overall AP. Code and demo are open-sourced at https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “Due to the lack of a value function for this policy, the affordances are implemented with a ViLD object detector [27].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “We find that two such models, ViLD [77] and MDETR [92], perform worse than humans but still quite resonably at providing Object feedback, even without domain-specific data for fine-tuning.”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Future work may explore open-vocabulary object detectors (Gu et al. 2021) as a means to recover salient details for more rich captions.”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “Similarly, some open-vocabulary detection models directly leverage foundation models for distillation or transfer like OV-DETR [54] and VILD [13].”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “ViLD [13] Mask R-CNN [15] X sentence X partial label partial label RegionCLIP [62] Faster RCNN [39] X sentence X partial label partial label FindIt [21] Faster RCNN [39] A sentence X partial label fine-tune MDETR [18] DETR [2] A,C word fine-tune zero-shot fine-tune DQ-DETR [46] DETR [2] A,C word X z”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “Many recent works aim to transfer the language capabilities of these models to object detection [12,26,45,46,20].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “3) End-to-end Task Planners with 3D Vision: Some task planners also explore the use of 3D vision.”
LERF: Language Embedded Radiance Fields (2023, methodology): “OpenSeg [14] simultaneously learns a mask prediction model while predicting the text embeddings for each mask, while ViLD [16] directly uses CLIP [28] to classify 2D regions from class-agnostic mask proposal networks.”
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, background): “Recently, large-scale pretrained vision-language models (VLMs) like CLIP [38] or ViLD [21] have been applied to robotics for learning representations and scene descriptors.”
Text2Motion: from natural language instructions to feasible plans (2023, methodology): “Suitable examples could be scene graph generation methods [43, 27] or forms of text-based scene description [22, 35, 58].”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “Image-language aligned models like CLIP [6] and ALIGN [9] encode text and images into a shared concept space, and have driven progress in several open-set tasks [11, 12, 13, 14, 15, 16, 17, 18].”
DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment (2023, methodology): “We focus on the GLIP protocol [27] rather than the ViLD protocol [16] that splits LVIS into seen/unseen categories, since the former is a stronger and more practical open-world setting that does not make any prior assumptions on downstream tasks while the latter still requires partial LVIS data for ”
Exploring CLIP for Assessing the Look and Feel of Images (2022, background): “…with massive image-text pairs, CLIP demonstrates exceptional capability in building semantic relationship between texts and visual entities without explicit training (Patashnik et al. 2021; Jain, Tancik, and Abbeel 2021; Hessel et al. 2021; Rao et al. 2022; Zhou, Loy, and Dai 2022; Gu et al. 2022).”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, background): “Various downstream vision tasks, e.g ., text-driven image manipulation [42], image captioning [25], view synthesis [30], and object detection [19], have attempted to exploit such features for improved generality and robustness.”
Code as Policies: Language Model Programs for Embodied Control (2022, methodology): “ViLD is used for the mobile robot domain, while MDETR is used for the tabletop manipulation and whiteboard drawing domains.”
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective (2023, neutral): “Shortly after its inception, CLIP has found its way to (open-vocabulary) semantic segmentation [23, 68] and object detection [15, 26].”