Dynamic Multimodal Instance Segmentation guided by natural language queries

By Edgar Margffoy-Tuay, Juan Pérez, Emilio Botero, Pablo Arbeláez. 2018

201 citations recorded by the source corpus.

We address the problem of segmenting an object given a natural language expression that describes it. Current techniques tackle this task by either (i) directly or recursively merging linguistic and visual information in the channel dimension and then performing convolutions; or by (ii) mapping the expression to a space in which it can be thought of as a filter, whose response is directly related to the presence of the object at a given spatial coordinate in the image, so that a convolution can be applied to look for the object. We propose a novel method that integrates these two insights in order to fully exploit the recursive nature of language. Additionally, during the upsampling process, we take advantage of the intermediate information generated when downsampling the image, so that detailed segmentations can be obtained. We compare our method against the state-of-the-art approaches in four standard datasets, in which it surpasses all previous methods in six of eight of the splits for this task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, background): “Models are usually designed specifically to learn from target datasets using various multimodal fusion strategies [29, 47, 53, 75, 83, 86].”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “Therefore, we cast the referring video object segmentation task as a joint problem of referring object segmentation in an image [12, 18, 36, 37] and mask propagation in a video [32, 22].”
Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation (2020, background): “Referring Expression Comprehension (REC) [11, 12, 19, 21, 44, 45, 48, 42, 37] and Referring Expression Segmentation (RES) [32, 16, 40, 25, 34] are two emerging tasks, which involves identifying the target visual instances according to a given linguistic expression.”
GRES: Generalized Referring Expression Segmentation (2023, methodology): “Methods Visual Encoder Textual Encoder RefCOCO RefCOCO+ G-Ref val test A test B val test A test B val(U) test(U) val(G) DMN [18] DPN92 SRU 49.”
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation (2021, background): “Previous approaches have developed various mechanisms for addressing this challenge, including dynamic convolutions [43], concatenations [21,30, 43], cross-modal attentions [15,23,40,50,62], graph neural networks [34], etc .”
Zero-Shot Semantic Segmentation via Variational Mapping (2019, background): “In addition, there are some studies on segmentation referring expressions like a keyword or sentence [27, 14, 12, 18].”
Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, background): “, [16, 31]) and train it to output the segmented masks [18, 29, 38, 53, 56].”
Segment Everything Everywhere All at Once (2023, background): “Existing ways of interactive segmentation are either using spatial hints like clicks or scribbles [7, 32] or referring segmentation using language [16, 36, 17, 65].”
PhraseClick: Toward Achieving Flexible Interactive Segmentation by Phrase and Click (2020, methodology): “Methods RMI [35] DMN [42] RNN [30] MAttNet [61] CMSA [60] Phrase PhraseClick”
A Fast and Accurate One-Stage Approach to Visual Grounding (2019, background): “We contrast our approach to some closely related works, including two existing one-stage grounding methods [52, 44] and some on grounding as segmentation [13, 20, 25, 9].”
Vision-Language Transformer and Query Generation for Referring Segmentation (2021, unknown): “Currently, the Fully Convolutional Network (FCN)-like framework [17, 5, 6, 4] is widely used in referring segmentation methods [10, 22].”
FindIt: Generalized Localization with Natural Language Queries (2022, background): “The research community has studied visual grounding through tasks including phase grounding, object retrieval and localization, language-driven instance segmentation, and others [64,72,62,70,57,82,26,22].”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, background): “Our language-conditioned segmentation problem is a variant of referring expression recognition [31, 32, 33, 34, 35].”