Simple Open-Vocabulary Object Detection with Vision Transformers

By Matthias Minderer, Alexey A. Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen and 4 more. 2022

421 citations recorded by the source corpus.

Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “ViLD [13] Mask R-CNN [15] X sentence X partial label partial label RegionCLIP [62] Faster RCNN [39] X sentence X partial label partial label FindIt [21] Faster RCNN [39] A sentence X partial label fine-tune MDETR [18] DETR [2] A,C word fine-tune zero-shot fine-tune DQ-DETR [46] DETR [2] A,C word X z”
Generalized Decoding for Pixel, Image, and Language (2022, background): “VL pre-trained models can be further extended to region-level tasks, such as phrase grounding and open-vocabulary object detection [25, 33, 54, 91], and unified frameworks that aim to combine image-text pairs with region-level data have also been proposed [4, 20, 42, 82, 89].”
Training-Free Layout Control with Cross-Attention Guidance (2023, background): “In the simplest case, given the text prompt y , the index i of a word y i in the text prompt, and a bounding box B , we would like to generate an image x that contains y i inside B , essentially modifying the generator to sample from a new distribution p ( x | y, B, i ) with additional controls.”
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “Then, we replace the visual search mechanism with open-world detectors GroundingDINO [29] and OWL-ViT [34] and use the detection results to fill in the VWM.”
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, neutral): “3 ViT-L/14 (303M) ViT-L/14 OWL-ViT (Minderer et al., 2022) 7 3 25.”
OpenOOD: Benchmarking Generalized Out-of-Distribution Detection (2022, background): “Future Work In the future, apart from maintaining the codebase, it is promising to extend our benchmark towards more robust OOD detectors [82] and object-level OOD detection and generalization [69, 83, 84, 85, 86, 87], which provides finer-grained visual guidance in safety-critical applications, suc”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “For example, we can switch to object detectors that are more robust and open-vocabulary, such as OWL-ViT (Minderer et al., 2022).”
A Cookbook of Self-Supervised Learning (2023, background): “These features are also leveraged by recent works for open-vocabulary object detection [Kamath et al., 2021, Gu et al., 2021, Zareian et al., 2021, Minderer et al., 2022].”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “OWL-ViT [206] [code] Propose ViT-based open-vocabulary detector by adding object classification/localization head.”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “Thus, there can be ambiguities in the choice of the arm depending on the task; e.g., Fig.”
Scaling Open-Vocabulary Object Detection (2023, methodology): “a more extreme version of the mosaics in [21]).”
LERF: Language Embedded Radiance Fields (2023, methodology): “We also provide quantitative evaluations against popular open-vocab detectors LSeg [21] and OWL-ViT [25], by distilling LSeg features into 3D [20] and querying OWL-ViT from rendered novel views.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “…[1, 2, 3] but emergent problem-solving [4, 5, 6] and creative generation of prose [7, 2] and code [8], while vision-language models enable open-vocabulary visual recognition [9, 10, 11] and can even make complex inferences about object-agent interactions in images [12, 13, 14, 15, 16, 17, 18].”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, methodology): “We use OWL-ViT [135] for open-vocabulary object detection, SAM [132] for initial-frame segmentation, and Cutie [136] for mask tracking.”
Foundation models in robotics: Applications, challenges, and the future (2023, positive): “…Radosavovic et al. [49], LATTE [50] R e l e v a n t t o R obo ti c s Perception Open-VocabularyObjectDetection and3Dclassiﬁcation e.g. OWL-ViT [51], GLIP [52], Grounding DINO [53], PointCLIP [54], PointBERT [55], ULIP [56], [57] Open-Vocabulary Semantic Segmentation e.g. LSeg [58], Segment…”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “Real : SayCan setup LEO [159] ConvNeXt [160], PointNet++ [161] Vicuna-7B Concat MLP Yes Sim-Mani : CLIPort task subset; Sim-Navi : Habitat-web 3D-LLM [57] CLIP, 3D-CLR [162], ConceptFusion [163] Flamingo, BLIP-2”