TransVG: End-to-End Visual Grounding with Transformers

By Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wen-gang Zhou, Houqiang Li. 2021

494 citations recorded by the source corpus.

In this paper, we present a neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image. The state-of-the-art methods, including two-stage or one-stage ones, rely on a complex module with manually-designed mechanisms to perform the query reasoning and multi-modal fusion. However, the involvement of certain mechanisms in fusion module design, such as query decomposition and image scene graph, makes the models easily overfit to datasets with specific scenarios, and limits the plenitudinous interaction between the visual-linguistic context. To avoid this caveat, we propose to establish the multi-modal correspondence by leveraging transformers, and empirically show that the complex fusion modules (e.g., modular attention network, dynamic graph, and multi-modal tree) can be replaced by a simple stack of transformer encoder layers with higher performance. Moreover, we re-formulate the visual grounding as a direct coordinates regression problem and avoid making predictions out of a set of candidates (i.e., region proposals or anchor boxes). Extensive experiments are conducted on five widely used datasets, and a series of state-of-the-art records are set by our TransVG. We build the benchmark of transformer-based visual grounding framework and make the code available at https://github.com/djiajunustc/TransVG.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “The key to open-set detection is introducing language for unseen object generalization [1, 7, 26].”
Transformers in Vision: A Survey (2021, methodology): “TransVG [204] is a simple design, where visual and text features are fused together in a transformer module, and the boundingbox corresponding to the query is directly regressed using a learnable token (input to the Transformer module, along-with visual and text features).”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, background): “tions, phrase grounding can be reformulated as a REC task by extracting phrases as referring expressions since phrases are assumed known during test (Deng et al. 2021; Du et al. 2021; Huang et al. 2022).”
Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling (2021, methodology): “The uni-ﬁed text+box output in U NICORN presents a more natu-ral way of visual grounding, compared to the box regression [14,24,62] or region index classiﬁcation [9,10,63].”
A Survey of Visual Transformers (2021, methodology): “Following ViT [29], TransVG [155] keeps the class token to aggregate the image and language features simultaneously for the mentioned object localization in REC.”
FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “We note that it is unclear how to use existing grounding models out-of-the-box for task unification due to the task-specific architectures, losses, and training strategies [13,21,51,55,88,89,91].”
UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling (2021, methodology): “Moreover, UniTAB’s unified output with both text and box presents a more natural way of visual grounding, compared to box regression [70,17,31] or region index classification [71,12,13].”
Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, methodology): “Importantly, unlike [7, 10, 23] our approach formulated in multi-task setting and solves both REC and RES tasks simultaneously; while the aforementioned works focus on REC specifically.”
Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, methodology): “Recent advancements in REC can be divided into two research directions: two-stage methods [3, 17, 18, 29, 54] and one-stage methods [8, 25, 27, 35, 51, 52].”