Recurrent Multimodal Interaction for Referring Image Segmentation

By Chenxi Liu, Zhe L. Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Alan L. Yuille. 2017

305 citations recorded by the source corpus.

In this paper we are interested in the problem of image segmentation given natural language descriptions, i.e. referring expressions. Existing works tackle this problem by first modeling images and sentences independently and then segment images by combining these two types of representations. We argue that learning word-to-image interaction is more native in the sense of jointly modeling two modalities for the image segmentation task, and we propose convolutional multimodal LSTM to encode the sequential interactions between individual words, visual information, and spatial information. We show that our proposed model outperforms the baseline model on benchmark datasets. In addition, we analyze the intermediate output of the proposed multimodal LSTM approach and empirically explain how this approach enforces a more effective word-to-image interaction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, background): “[49] Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille.”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “37 D+RMI+DCRF [14] res101-DeepLab testB 44.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Existing work in visual grounding [38, 30, 25, 40, 35] has addressed the problem of passively perceiving a static image and mapping a referential expression to a bounding box [38, 30, 25] or a segmentation mask [24, 27, 55], exploring various techniques including proposal generation [10] and relatio”
Referring Image Segmentation via Recurrent Refinement Networks (2018, methodology): “Baseline We re-implement the D-LSTM model [23] with the help of authors’ code that is publicly available.”
Image Segmentation Using Text and Image Prompts (2021, background): “Early approaches used recurrent networks in combination with CNNs to address this problem [15, 16, 17, 18].”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “A na¨ıve baseline for the task is applying referring image segmentation techniques [16, 12, 37, 36] to each input frame independently.”
Cross-Modal Self-Attention Network for Referring Image Segmentation (2019, methodology): “[17, 20]) focus on learning multimodal interaction in a sequential manner.”
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation (2021, methodology): “Concretely, the fusion strategies include recurrent interaction [30,33], cross-modal attention [5,23,50], multi-1 modal graph reasoning [24], linguistic structure-guided context modeling [25], etc .”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “Large amounts of work have been done to learn across image and language, including visualgrounding [16,81,49,96,91,62], image captioning [6,1,5], visual question and answering (VQA) [33,12], visual dialog [11,39], visual reasoning [75,93,85], text-toimage and image-to-text retrieval [32,66,74], and ”
Improving One-stage Visual Grounding by Recursive Sub-query Construction (2020, methodology): “The per-word sub-query modeling is used by RMI [23] for referring image segmentation.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2016) VGG LSTM Context 7 (Luo & Shakhnarovich, 2017) VGG BiLSTM CG ADAM 7 (Liu et al., 2017) VGG LSTM Combined ADAM 7 (Hu et al.”
Dynamic Multimodal Instance Segmentation guided by natural language queries (2018, methodology): “roceed to compare our full method with the state-of-the-art, for which we evaluate on all the datasets described above. Table2compares the mIoU of our method with the state-of-the-art in this task [3][4][5]. The results show that our method outperforms all other methods in six out of eight splits of”
PhraseClick: Toward Achieving Flexible Interactive Segmentation by Phrase and Click (2020, methodology): “We propose to utilize both interaction types to make selection ﬂexible and robust to handle complex scenarios. to drive the selection [24,35,50,61].”
Segment Everything Everywhere All at Once (2023, background): “…segmentation that segments an image thoroughly given a predetermined set of concepts, language-based referring segmentation provides a user-friendly way of segmenting a speciﬁc region referred by an arbitrary text phrase [13, 14, 15, 16, 17]; and 3) From one-shot to interactive segmentation.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, methodology): “In contrast, other methods [19, 39, 63] address the referring expression task with a single stage proposing end-to-end approaches, and some [19, 32, 30, 36, 64, 4] produce a segmentation mask for the object description.”
CRIS: CLIP-Driven Referring Image Segmentation (2021, methodology): “Benefiting from the powerful capacity of the deep neural network, early approaches [14, 22, 25, 33] concatenate textual features with each visual activation directly, and use these combined features to generate the segmentation mask.”
Scene Graph Parsing as Dependency Parsing (2018, background): “…2015; Vendrov et al., 2015), image captioning (Mao et al., 2014; Karpathy and Li, 2015; Donahue et al., 2015; Liu et al., 2017b), visual question answering (Antol et al., 2015; Zhu et al., 2016; Andreas et al., 2016), and referring expressions (Hu et al., 2016; Mao et al., 2016; Liu et al., 2017a).”
Vision-Language Transformer and Query Generation for Referring Segmentation (2021, methodology): “ the linguistic features extracted by LSTM and vision features extracted by CNN. Then, a fully convolutional network (FCN) is applied on the fused feature, proving the feasibility of this problem. In [16], in order to make use of each word in the referring sentence, Liu et al. proposed a recurrent n”