Segmentation from Natural Language Expressions

By Ronghang Hu, Marcus Rohrbach, Trevor Darrell. 2016

568 citations recorded by the source corpus.

In this paper we approach the novel problem of segmenting an image based on a natural language expression. This is different from traditional semantic segmentation over a predefined set of semantic classes, as e.g., the phrase “two men sitting on the right bench” requires segmenting only the two people on the right bench and no one standing or sitting on another bench. Previous approaches suitable for this task were limited to a fixed set of categories and/or rectangular regions. To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information. In our model, a recurrent neural network is used to encode the referential expression into a vector representation, and a fully convolutional network is used to a extract a spatial feature map from the image and output a spatial response map for the target object. We demonstrate on a benchmark dataset that our model can produce quality segmentation output from the natural language expression, and outperforms baseline methods by a large margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
Learning to Reason: End-to-End Module Networks for Visual Question Answering (2017, methodology): “To help reason about spatial properties, we add two extra x = i 15 and y = j 10 dimensions to each location (i, j) on the feature map similar to [13], so the final visual feature xvis on each image is a 15× 10× 514 tensor.”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “Our model can also be used to address referential object segmentation [8, 14].”
Natural Language Object Retrieval (2015, background): “As follow up to this work we show successful results by encoding the phrase rather than scoring it [27] and also predicting image segmentations instead of bounding boxes [14].”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Existing work in visual grounding [38, 30, 25, 40, 35] has addressed the problem of passively perceiving a static image and mapping a referential expression to a bounding box [38, 30, 25] or a segmentation mask [24, 27, 55], exploring various techniques including proposal generation [10] and relatio”
Modeling Relationships in Referential Expressions with Compositional Modular Networks (2016, background): “9 over all 25 locations on the grid, and language representation qloc obtained by scanning through the word embedding sequence with a single LSTM network and taking the hidden state at the last time step same as in [24, 9].”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Vision-and-Language Grounding Recently, researchers in both computer vision and natural language processing are striving to bridge vision and natural language towards a deeper understanding of the world [46, 40, 20, 6, 24, 17, 37, 19], e.”
Guide Me: Interacting with Deep Networks (2018, methodology): “Other examples include generation of referring expressions [36, 44, 47, 62], segmentation or object retrieval from referring expressions [19, 29, 28], image captioning [16, 21, 33, 34, 48, 56] and visual dialog [18].”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “[87] developed a semantic segmentation algorithm that combines a CNN to encode the image and an LSTM to encode its linguistic description.”
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (2016, background): “…techniques (Krishnamurthy and Kollar, 2013; Matuszek et al, 2012), there is a recent trend of machine learning-based approaches (Karpathy and Fei-Fei, 2015; Karpathy et al, 2014; Akata et al, 2016; Kong et al, 2014; Hu et al, 2016; Rohrbach et al, 2015a; Mao et al, 2016) to find the associations.”
Zero-Shot Semantic Segmentation via Variational Mapping (2019, background): “In addition, there are some studies on segmentation referring expressions like a keyword or sentence [27, 14, 12, 18].”
Referring Image Segmentation via Recurrent Refinement Networks (2018, methodology): “The two resulting features are concatenated with spatial coordinates [9] or multimodal information [23], and passed through pixel-wise classification networks to produce corresponding masks matching language expressions.”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, background): “, text-to-image retrieval [44], image caption [48], visual question answering [1], referring segmentation [19,49,50] and so on.”
Segment Everything Everywhere All at Once (2023, background): “Existing ways of interactive segmentation are either using spatial hints like clicks or scribbles [7, 32] or referring segmentation using language [16, 36, 17, 65].”
Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation (2020, methodology): “Referring Expression Comprehension (REC) [11, 12, 19, 21, 44, 45, 48, 42, 37] and Referring Expression Segmentation (RES) [32, 16, 40, 25, 34] are two emerging tasks, which involves identifying the target visual instances according to a given linguistic expression.”
VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation (2017, methodology): “In light of the premises and demerits of image captioning, visual question answering (VQA) [3, 51, 36, 13] and visual grounding [35, 15, 37, 28, 16, 43, 50] are proposed, in parallel, to accommodate automatic evaluation and multiple levels of focus on the visual entities (e.g., scene, object,…”