Improving One-stage Visual Grounding by Recursive Sub-query Construction

By Zhengyuan Yang, Tianlang Chen, Liwei Wang, Jiebo Luo. European Conference on Computer Vision, 2020

310 citations recorded by the source corpus.

We improve one-stage visual grounding by addressing current limitations on grounding long and complex queries. Existing one-stage methods encode the entire language query as a single sentence embedding vector, e.g., taking the embedding from BERT or the hidden state from LSTM. This single vector representation is prone to overlooking the detailed descriptions in the query. To address this query modeling deficiency, we propose a recursive sub-query construction framework, which reasons between image and query for multiple rounds and reduces the referring ambiguity step by step. We show our new one-stage method obtains 5.0%, 4.5%, 7.5%, 12.8% absolute improvements over the state-of-the-art one-stage baseline on ReferItGame, RefCOCO, RefCOCO+, and RefCOCOg, respectively. In particular, superior performances on longer and more complex queries validates the effectiveness of our query modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f79e8810f71889ca24a69c7cc6cf398ff2adf134

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, background): “The other one is single-stage modeling, [65, 76, 41, 64], which considers REC as a language-guided detection task.”
Visual Grounding with Transformers (2021, methodology): “Following previous works [46, 45], we use the Precision”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, background): “Most bottom-up solutions (Chen et al. 2018; Liao et al. 2019; Miao et al. 2022; Yang et al. 2019, 2020) fuse multi-modality features and output the ﬁnal boxes directly.”
GRES: Generalized Referring Expression Segmentation (2023, methodology): “[13], Referring Expression Segmentation (RES) comes from a similar task, Referring Expression Compre-hension (REC) [14, 25, 29, 40, 44, 45, 54] that outputs a bounding box for the target.”
Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, methodology): “[51, 52] proposed to fuse query information with visual features and pick the bounding box with maximum activation scores from YOLOv3 [43].”
FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “We note that it is unclear how to use existing grounding models out-of-the-box for task unification due to the task-specific architectures, losses, and training strategies [13,21,51,55,88,89,91].”
TransVG: End-to-End Visual Grounding with Transformers (2021, methodology): “By replacing ResNet-50 with a stronger ResNet-101, the performance boosts to 70.73%, which is 6.13% higher than the strongest competitor ReSC-Large for one-stage methods and 7.73% higher than the strongest competitor DDPN for two-stage methods, respectively.”