Vision-Language Transformer and Query Generation for Referring Segmentation

By Henghui Ding, Chang Liu, Suchen Wang, Xudong Jiang. IEEE International Conference on Computer Vision, 2021

380 citations recorded by the source corpus.

In this work, we address the challenging task of referring segmentation. The query expression in referring segmentation typically indicates the target object by describing its relationship with others. Therefore, to find the target one among all instances in the image, the model must have a holistic understanding of the whole image. To achieve this, we reformulate referring segmentation as a direct attention problem: finding the region in the image where the query language expression is most attended to. We introduce transformer and multi-head attention to build a network with an encoder-decoder attention mechanism architecture that "queries" the given image with the language expression. Furthermore, we propose a Query Generation Module, which produces multiple sets of queries with different attention weights that represent the diversified comprehensions of the language expression from different aspects. At the same time, to find the best way from these diversified comprehensions based on visual clues, we further propose a Query Balance Module to adaptively select the output features of these queries for a better mask generation. Without bells and whistles, our approach is light-weight and achieves new state-of-the-art performance consistently on three referring segmentation datasets, RefCOCO, RefCOCO+, and G-Ref. Our code is available at https://github.com/henghuiding/Vision-Language-Transformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/367a5169efb790a15bcc1afcb0164b32740b751a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Early methods [17], [18] combine the self-attention layers to augment CNNs.”
CRIS: CLIP-Driven Referring Image Segmentation (2021, methodology): “On the RefCOCO dataset, our model significantly outperforms the state-of-the-art Vision Language Transformer [5] by 4.”
GRES: Generalized Referring Expression Segmentation (2023, methodology): “Most recently, some transformer-based methods [5, 6, 39, 44] are proposed and bring large performance gain compared to the CNN-based network.”
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels (2021, methodology): “, RefCOCO [56]) for this task, previously developed methods are usually fully supervised [21,55,12,22,54].”
LViT: Language Meets Vision Transformer in Medical Image Segmentation (2022, background): “Subsequently, there is a rich line of works on image segmentation [21], [27], [28], [29], [30], [31] that have begun to use text information to improve the segmentation capabilities of models.”
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation (2021, methodology): “Compared with the second-best method, VLT [13], LAVT achieves higher performance with absolute margins of 7.08%, 7.53%, and 6.06% on the validation, testA, and testB subsets of RefCOCO, respectively.”