A Fast and Accurate One-Stage Approach to Visual Grounding

By Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, Jiebo Luo. 2019

449 citations recorded by the source corpus.

We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight. The performances of existing propose-and-rank two-stage methods are capped by the quality of the region candidates they propose in the first stage --- if none of the candidates could cover the ground truth region, there is no hope in the second stage to rank the right region to the top. To avoid this caveat, we propose a one-stage model that enables end-to-end joint optimization. The main idea is as straightforward as fusing a text query's embedding into the YOLOv3 object detector, augmented by spatial features so as to account for spatial mentions in the query. Despite being simple, this one-stage approach shows great potential in terms of both accuracy and speed for both phrase localization and referring expression comprehension, according to our experiments. Given these results along with careful investigations into some popular region proposals, we advocate for visual grounding a paradigm shift from the conventional two-stage methods to the one-stage framework.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “, image-text retrieval [60], visual question answering [6], visual grounding [28, 58], etc.”
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (2021, background): “Others [16, 22] first merge all the ground-truth boxes associated to the phrase by considering the smallest enclosing box.”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “Moreover, UniTAB’s unified output with both text and box presents a more natural way of visual grounding, compared to box regression [70,17,31] or region index classification [71,12,13].”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, background): “Sundry efforts that have made progress towards binding the two fields include [8, 5, 22, 32] in visual grounding, [31, 30, 36, 25] in image/video captioning, [10, 2, 35, 37] in video moment retrieval, and”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “object with designated description in a given image [30], [31].”
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, background): “The other one is single-stage modeling, [65, 76, 41, 64], which considers REC as a language-guided detection task.”
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks (2022, background): “This is different from previous multi-modal object detection frameworks [18, 52], where the detection results depend on both image and language inputs.”
GRES: Generalized Referring Expression Segmentation (2023, methodology): “The earliest dataset for RES and REC is ReferIt [20] , in which one expression only refers to one instance.”
Improving One-stage Visual Grounding by Recursive Sub-query Construction (2020, methodology): “There are mainly two threads of works in visual grounding: the two-stage approach [41,40,32,3,50,48] and one-stage approach [47,5,37].”
Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, methodology): “[51, 52] proposed to fuse query information with visual features and pick the bounding box with maximum activation scores from YOLOv3 [43].”
Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark (2021, methodology): “It is a standard visual grounding task and we follow the algorithm [63] proposed by Yang et al.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, methodology): “In contrast, other methods [19, 39, 63] address the referring expression task with a single stage proposing endto-end approaches, and some [19, 32, 30, 36, 64, 4] produce a segmentation mask for the object description.”
FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “We note that it is unclear how to use existing grounding models out-of-the-box for task unification due to the task-specific architectures, losses, and training strategies [13,21,51,55,88,89,91].”
Visual Grounding with Transformers (2021, methodology): “The proposal-free methods [7, 9, 16, 10] focus on localizing the referred object directly without proposals.”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, methodology): “2017), FAOG (Yang et al. 2019), SimNet-CCA (Plummer et al.”
Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling (2021, methodology): “The uni-ﬁed text+box output in U NICORN presents a more natu-ral way of visual grounding, compared to the box regression [14,24,62] or region index classiﬁcation [9,10,63].”
Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation (2020, methodology): “To speedup the inference, some recent works in REC resort to a one-stage modeling [33, 38], which embeds the extracted linguistic feature into a one-stage detection network, e.”
Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, methodology): “Recently, one-stage methods [8, 25, 27, 35, 51, 52] that directly predict the target box have been used to alleviate the limitations of the aforementioned two-stage methods.”