Visual Grounding with Transformers

By Ye Du, Zehua Fu, Qingjie Liu, Yunhong Wang. 2021

86 citations recorded by the source corpus.

In this paper, we propose a transformer based approach for visual grounding. Unlike existing proposal-and-rank frameworks that rely heavily on pretrained object detectors or proposal-free frameworks that upgrade an off-the-shelf one-stage detector by fusing textual embeddings, our approach is built on top of a transformer encoder-decoder and is independent of any pretrained detectors or word embedding models. Termed as VGTR – Visual Grounding with TRansformers, our approach is designed to learn semantic-discriminative visual features under the guidance of the textual description without harming their location ability. This information flow enables our VGTR to have a strong capability in capturing context-level semantics of both vision and language modalities, rendering us to aggregate accurate visual clues implied by the description to locate the interested object instance. Experiments show that our method outperforms state-of-the-art proposal-free approaches by a considerable margin on four benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fahad Shahbaz Khan
2. Hang Su
3. Salman Hameed Khan
4. Mubarak Shah
5. Lei Zhang
6. Muzammal Naseer
7. Feng Li
8. Hao Zhang

In Computer Vision and Pattern Recognition

1. Fahad Shahbaz Khan
2. Luc Van Gool
3. Ming-Hsuan Yang
4. Salman Hameed Khan
5. Lei Zhang
6. Dacheng Tao
7. Bernt Schiele
8. Munawar Hayat

In Model Architectures

1. Fahad Shahbaz Khan
2. Feng Li
3. Hao Zhang
4. Lei Zhang
5. Hang Su
6. Jun-Juan Zhu
7. Leonid Sigal
8. Mubarak Shah

Citations in the corpus

• Transformers in Vision: A Survey (2021, methodology): “Visual Grounding with Transformer [58] has an encoder-decoder architecture, where visual tokens (features extracted from a pretrained CNN model) and text tokens (parsed through an RNN module) are processed in parallel with two distinct branches in the encoder, with cross-modality attention to genera”
• Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, methodology): “Compared to most state-of-the-art methods [8, 11, 22, 25, 47, 51] that only use textual features from the last layer of the language encoder, our method utilizes LAFM to adaptively generate textual features for visual features of each layer.”
• DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, background): “tions, phrase grounding can be reformulated as a REC task by extracting phrases as referring expressions since phrases are assumed known during test (Deng et al. 2021; Du et al. 2021; Huang et al. 2022).”
• Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, methodology): “Importantly, unlike [7, 10, 23] our approach formulated in multi-task setting and solves both REC and RES tasks simultaneously; while the aforementioned works focus on REC specifically.”
• A Survey of Visual Transformers (2021, methodology): “VGTR [156] reformulates the REC as a task for single salient object localization from the language features.”