Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
By Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun-yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu and 1 more. 2023
4,547 citations recorded by the source corpus.
In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Trevor Darrell
- Sergey Levine
- Yoshua Bengio
- Dacheng Tao
- Quoc V. Le
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Yu Qiao
- Jitendra Malik
- Andrew Zisserman
In Multimodal Learning
Citations in the corpus
- Visual Instruction Tuning (2023, background): “…foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44, 30].”
- Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “We will now use 3 × 3 convolutions.”
- Modular Visual Question Answering via Code Generation (2023, methodology): “We implement this function using Grounding DINO (Liu et al., 2023), which is an open-vocabulary object detector that is also trained on referring expression comprehension.”
- MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (2023, methodology): “…MiniGPT-v2 outperforms many VL-generalist models including VisionLLM [53], OFA [52] and Shikra [7] and reduces the accuracy gap comparing to specialist models including UNINEXT [54] and G-DINO [27]. caption → grounded phrases and phrase → grounded phrase, each containing around 2.5k and 3k samples.”
- ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, methodology): “We also develop a variant of our system, ConceptGraphs - Detector ( CG-D ), where we employ an image tagging model (RAM [55]) to list the object classes present in the image and an open-vocabulary 2D detector (Grounding DINO [34]) to obtain object bounding boxes 6 .”
- Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “For instance, VPEval [9] applies visual programming to compute image-text alignment, using ChatGPT to invoke expert VLMs like image captioning [40] and open-vocabulary detection [48] models to examine fine-grained visual details.”
- The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot (2023, methodology): “We begin by adapting the text-based approach of the combination of the GroundDINO [26] and SAM [19] methods to return the overall most probable object belonging to the specified class in its description.”
- OS-ATLAS: A Foundation Action Model for Generalist GUI Agents (2024, methodology): “Existing GUI grounding training data can be divided into two types: referring expression grounding (REG) (Liu et al., 2023) and instruction grounding (IG) (Li et al., 2020).”
- ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “We provide a solution to connect ImageBind-LLM with object detectors [50].”
- GLaMM: Pixel Grounding Large Multimodal Model (2023, methodology): “BuboGPT [59] utilizes an off-the-shelf grounding model [30] and matches the groundings with the language response.”
- InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, methodology): “Vision-language models (VLMs), exemplified by CLIP [66] and its subsequent works [26, 36, 43, 44, 50, 75, 94], align image and text features in a unified embedding space.”
- Robotic Control via Embodied Chain-of-Thought Reasoning (2024, neutral): “…is observed in robot datasets, the recent development of strong, open-source vision-language models that learn visuo-linguistic representations [28, 29], generate images from text [30], or generate text in response to images and prompts [31, 32, 33, 34, 35] have resulted in a large number of works…”
- Faster Segment Anything: Towards Lightweight SAM for Mobile Applications (2023, background): “[2022a], EfficientViT Liu et al. [2023b], Next-ViT Li et al. [2022b] and Tiny-ViT Wu et al. [2022]. The recent progress in lightweight and faster ViT is complementary to our proposed decoupled distillation towards making the next-generation SAM suitable for resource-constrained mobile devices.”
- V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “Then, we replace the visual search mechanism with open-world detectors GroundingDINO [29] and OWL-ViT [34] and use the detection results to fill in the VWM.”
- Visual-RFT: Visual Reinforcement Fine-Tuning (2025, neutral): “In reasoning grounding, Visual-RFT excels in the LISA [13] dataset, which heavily relies on reasoning, outperforming specialized models like GroundedSAM [20].”
- OmniGen: Unified Image Generation (2024, neutral): “The specific data format is shown in Figure 3-( A young woman stands on the platform of a railway station in spring.”
- VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation (2023, methodology): “Specifically, we utilize YOLOv7 [27] for target objects that fall within the COCO [28] classes, and Grounding-DINO [29] for all other categories.”
- Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, methodology): “Consequently, inspired by prior successful works such as Grounded Pre-training [81, 38] and SAM [25], we aim to address complex segmentation in the wild tasks by combining the strong open-set foundation models.”