Cross-Modal Self-Attention Network for Referring Image Segmentation

By Linwei Ye, Mrigank Rochan, Zhi Liu. 2019

588 citations recorded by the source corpus.

We consider the problem of referring image segmentation. Given an input image and a natural language expression, the goal is to segment the object referred by the language expression in the image. Existing works in this area treat the language expression and the input image separately in their representations. They do not sufficiently capture long-range correlations between these two modalities. In this paper, we propose a cross-modal self-attention (CMSA) module that effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the input image. In addition, we propose a gated multi-level fusion module to selectively integrate self-attentive cross-modal features corresponding to different levels in the image. This module controls the information flow of features at different levels. We validate the proposed approach on four evaluation datasets. Our proposed approach consistently outperforms existing state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, background): “[88] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang.”
Vision-Language Transformer and Query Generation for Referring Segmentation (2021, methodology): “This oblique process brings inefficiencies to information interactions among pixels/words in a distance [27], thus is undesirable for referring segmentation models to understand the global context of the image [28].”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “[72] introduced CMSA (Cross-Modal Self-Attention), which operates on the concatenation of visual features, word embed-dings, and spatial coordinates to model long-range dependencies between words and spatial regions.”
U-Net Transformer: Self and Cross Attention for Medical Image Segmentation (2021, unknown): “Recent applications include object detection [1], semantic segmentation [17, 14], and image classiﬁcation [4].”
Segment Everything Everywhere All at Once (2023, background): “…segmentation that segments an image thoroughly given a predetermined set of concepts, language-based referring segmentation provides a user-friendly way of segmenting a speciﬁc region referred by an arbitrary text phrase [13, 14, 15, 16, 17]; and 3) From one-shot to interactive segmentation.”
PhraseClick: Toward Achieving Flexible Interactive Segmentation by Phrase and Click (2020, methodology): “Referring expression comprehension methods [27, 35, 38, 54, 60–62] detect a speciﬁc object in an image given a referring expression.”
Cross-Modality Fusion Transformer for Multispectral Object Detection (2021, methodology): “Multimodal Transformers have been applied to various tasks including image segmentation [24, 25], cross-modal sequence generation[26, 27], video retrieval[28, 29] and image/video captioning/classification [30– 34].”
3D Human Pose Estimation with Spatial and Temporal Transformers (2021, background): “Moreover, existing vision transformers have been limited primarily to image classification [12, 36], object detection [4, 50], and segmentation [41, 47], but how to harness the power of transformers for 3D HPE remains unclear.”
Image Segmentation Using Text and Image Prompts (2021, methodology): “The CMSA module, which is central to the approach of Ye et al. [19], models long-term dependencies between text and image using attention.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, methodology): “In contrast, other methods [19, 39, 63] address the referring expression task with a single stage proposing end-to-end approaches, and some [19, 32, 30, 36, 64, 4] produce a segmentation mask for the object description.”
Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation (2020, methodology): “We further evaluate the proposed MCN on three benchmark datasets, i.e., RefCOCO [13], RefCOCO+ [13] and RefCOCOg [24], and compare them to a set of state-of-the-art methods [43, 37, 38, 40, 16] of both REC and RES.”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “However, a direct application of image-based solutions to referring object segmentation [12, 18, 36, 37] would fail to exploit valuable information, temporal coherence across the frames.”
Attention Bottlenecks for Multimodal Fusion (2021, methodology): “Multimodal transformers have been applied to various tasks including audio enhancement [17, 55], speech recognition [25], image segmentation [61, 55], cross-modal sequence generation [40, 38], video retrieval [21] and image/video captioning/classification [43, 54, 53, 37, 28].”
CRIS: CLIP-Driven Referring Image Segmentation (2021, methodology): “[48] designed a Cross-Modal Self-Attention (CSMA) module to focus on informative words in the sentence and crucial regions in the image.”
Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, background): “, [16, 31]) and train it to output the segmented masks [18, 29, 38, 53, 56].”
Deep Multimodal Data Fusion (2024, background): “For instance, the attention mechanism proposed by Ye et al. [211] adaptively shifts focus to salient words in the query expression and significant portions of the input image.”