F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

By Weicheng Kuo, Yin Cui, Xiuye Gu, A. Piergiovanni, Anelia Angelova. 2022

199 citations recorded by the source corpus.

We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models. F-VLM simplifies the current multi-stage training pipeline by eliminating the need for knowledge distillation or detection-tailored pretraining. Surprisingly, we observe that a frozen VLM: 1) retains the locality-sensitive features necessary for detection, and 2) is a strong region classifier. We finetune only the detector head and combine the detector and VLM outputs for each region at inference time. F-VLM shows compelling scaling behavior and achieves +6.5 mask AP improvement over the previous state of the art on novel categories of LVIS open-vocabulary detection benchmark. In addition, we demonstrate very competitive results on COCO open-vocabulary detection benchmark and cross-dataset transfer detection, in addition to significant training speed-up and compute savings. Code will be released at the https://sites.google.com/view/f-vlm/home

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Jianfeng Gao
3. Lei Zhang
4. Pieter Abbeel
5. Jianwei Yang
6. Ziwei Liu
7. Chen Change Loy
8. Trevor Darrell

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Jitendra Malik
5. Chen Change Loy
6. Ming-Hsuan Yang
7. Trevor Darrell
8. Li Fei-Fei

In Multimodal Learning

1. Jianfeng Gao
2. Jianwei Yang
3. Lei Zhang
4. Feng Li
5. Hao Zhang
6. Xueyan Zou
7. Chen Change Loy
8. Ziwei Liu

Citations in the corpus

• A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “In the light of strong zero-shot transferability demonstrated in the seminal work CLIP [39], a number of researchers have attempted to build advanced open-vocabulary models by leveraging largescale image-text pairs for fine-grained vision tasks like detection [10, 12, 22, 29, 56] and segmentation [7”
• Transformer-Based Visual Segmentation: A Survey (2023, methodology): “To handle this, Forzen-VLM [279] adopts the frozen visual clip model and combines the scores of both learned visual em-beddingandCLIPembeddingfornovelclassdetection.”
• SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “Despite the impressive advancements in Vision Language Models (VLMs) across a variety of tasks such as image classification [4], captioning [5], object detection [6], and document parsing [7], etc., these models still face significant challenges with spatial reasoning.”
• Language to Rewards for Robotic Skill Synthesis (2023, methodology): “In the real world, we detect objects in image-space using an open-vocabulary detector: F-VLM [53].”
• OpenScene: 3D Scene Understanding with Open Vocabularies (2022, background): “Many recent efforts [17,18,21,28,30,33,37,44,57,59,60] attempt to correlate the dense image features with the embedding from large language models.”