Referring Transformer: A One-step Approach to Multi-task Visual Grounding

By Muchen Li, Leonid Sigal. 2021

262 citations recorded by the source corpus.

As an important step towards visual reasoning, visual grounding (e.g., phrase localization, referring expression comprehension/segmentation) has been widely explored Previous approaches to referring expression comprehension (REC) or segmentation (RES) either suffer from limited performance, due to a two-stage setup, or require the designing of complex task-specific one-stage architectures. In this paper, we propose a simple one-stage multi-task framework for visual grounding tasks. Specifically, we leverage a transformer architecture, where two modalities are fused in a visual-lingual encoder. In the decoder, the model learns to generate contextualized lingual queries which are then decoded and used to directly regress the bounding box and produce a segmentation mask for the corresponding referred regions. With this simple but highly contextualized model, we outperform state-of-the-arts methods by a large margin on both REC and RES tasks. We also show that a simple pre-training schedule (on an external dataset) further improves the performance. Extensive experiments and ablations illustrate that our model benefits greatly from contextualized information and multi-task training.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Fahad Shahbaz Khan
2. Salman Hameed Khan
3. Luc Van Gool
4. Ming-Hsuan Yang
5. Lei Zhang
6. Munawar Hayat
7. Dacheng Tao
8. Syed Waqas Zamir

In Multimodal Learning

1. Mubarak Shah
2. Siyi Liu
3. Salman Hameed Khan
4. Hang Su
5. Jianping Fan
6. Feng Li
7. Hao Zhang
8. Fahad Shahbaz Khan

In Model Architectures

1. Fahad Shahbaz Khan
2. Feng Li
3. Hao Zhang
4. Lei Zhang
5. Hang Su
6. Jun-Juan Zhu
7. Mubarak Shah
8. Munawar Hayat

Citations in the corpus

• Transformers in Vision: A Survey (2021, methodology): “Referring Transformer [121] is also a simple one stage design where the text and image features are fused in a Transformer encoder, and the Transformer-based decoder then directly regresses bounding boxes or segmentation masks.”
• Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, methodology): “MCN [35] and RT [25] are multi-task learning methods including REC and RES while PFOS [47] is multi-task learning method including REC and REG.”
• DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, methodology): “…We compare our DQ-DETR with BAN (Kim, Jun, and Zhang 2018), VisualBert (Li et al. 2019), CITE (Plummer et al. 2017), FAOG (Yang et al. 2019), SimNet-CCA (Plummer et al. 2020), DDPN (Yu et al. 2018b), RefTR (Li and Sigal 2021), SeqTR (Zhu et al. 2022), and MDETR (Kamath et al. 2021) in Table 3.”
• A Survey of Visual Transformers (2021, methodology): “Visual Grounding: MDETR [154], TransVG [155], VGTR [156], Referring Transformer [157], Pseudo-Q [158], LanguageRefer [159],”