Natural Language Object Retrieval

By Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, Trevor Darrell. 2015

576 citations recorded by the source corpus.

In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object. Natural language object retrieval differs from text-based image retrieval task as it involves spatial information about objects within the scene and global scene context. To address this issue, we propose a novel Spatial Context Recurrent ConvNet (SCRC) model as scoring function on candidate boxes for object retrieval, integrating spatial configurations and global scene-level contextual information into the network. Our model processes query text, local image descriptors, spatial configurations and global context features through a recurrent network, outputs the probability of the query text conditioned on each candidate box as a score for the box, and can transfer visual-linguistic knowledge from image captioning domain to our task. Experimental results demonstrate that our method effectively utilizes both local and global information, outperforming previous baseline methods significantly on different datasets and scenarios, and can exploit large scale vision and language datasets for knowledge transfer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Learning Visual Representations with Caption Annotations (2020, background): “Vision and language (VL) have been jointly leveraged to learn cross-modal representations for various VL tasks, such as crossmodal retrieval [21,61], visual question answering [25], captioning [56] or visual grounding [11,30].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, positive): “nces results in much better performance than weakly supervised training, thus conﬁrming the usefulness of our annotations. Since then, a number of other works have adopted Flickr30K Entities as well. Hu et al. (2016) leverage spatial information and global context to model where objects are likely t”
Neural Module Networks (2015, background): “This grounding task has previously been approached in [18, 32, 17, 20, 14], where the authors tried to localize phrases in an image.”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, background): “Similarly, Hu et al. (2016a) concatenate phrase embeddings with visual features at different spatial locations to obtain a segmentation.”
Bilinear Attention Networks (2018, neutral): “Model Detector R@1 R@5 R@10 Upper Bound Zhang et al. [40] MCG [3] 28.5 52.7 61.3 - Hu et al. [11] Edge Boxes [42] 27.8 - 62.9 76.9 Rohrbach et al. [26] Fast RCNN [7] 42.43 - - 77.90 Wang et al. [33] Fast RCNN [7] 42.08 - - 76.91 Wang et al. [32] Fast RCNN [7] 43.89 64.46 68.66 76.91 Rohrbach et al.…”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, neutral): “Visual information retrieval with natural language queries has long been tackled via joint visuallanguage embedding models [12,19,24,25,26,27,28].”
TVQA: Localized, Compositional Video Question Answering (2018, background): “For image-based object grounding, there has been much work on phrase grounding (Plummer et al., 2015; Wang et al., 2016b; Rohrbach et al., 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “Most recent work on referring expressions uses CNN-LSTM based frameworks to model P (r|o) [18, 10, 31, 19, 17] or uses a joint vision-language embedding”
Generation and Comprehension of Unambiguous Object Descriptions (2015, methodology): “Concurrently, [24] propose a CNN-RNN based method that is similar to our baseline model and achieve state-ofthe-art results on the ReferIt dataset [27].”
Localizing Moments in Video with Natural Language (2017, methodology): “For every tenth frame in a video, we score candidate bounding boxes with the object retrieval model proposed in [14] and compute the score for a frame as the maximum score of all bounding boxes.”
Grounding of Textual Phrases in Images by Reconstruction (2015, methodology): “n- and semi-supervised training regimes. We believe that our approach of encoding the phrase optimizes the better objective for grounding than scoring the phrase with a text generation pipeline as in [18,33]. As for the fully-supervised regime we empirically show our advantage over [18]. [36] attemp”
Dual Attention Networks for Multimodal Reasoning and Matching (2016, background): “Recently, dramatic advances in deep learning have broken the boundaries between vision and language, drawing growing interest in their intersection, such as visual question answering (VQA) [3, 37, 23, 35], image captioning [33, 2], image-text matching [8, 11, 20, 30], visual grounding [24, 9], etc.”
Modeling Context in Referring Expressions (2016, background): “Three recent approaches for referring expression generation [26] and comprehension [14,33] also take a deep learning approach.”
Neural Baby Talk (2018, background): “Another related line of work is on resolving referring expressions [21] (or description-based object retrieval [36, 17, 18, 39] – given a description of an object in the image, identify which object is being referred to) or referring expression generation [21, 29, 31, 50] (given an object in the ima”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Existing work in visual grounding [38, 30, 25, 40, 35] has addressed the problem of passively perceiving a static image and mapping a referential expression to a bounding box [38, 30, 25] or a segmentation mask [24, 27, 55], exploring various techniques including proposal generation [10] and relatio”
GuessWhat?! Visual Object Discovery through Multi-modal Dialogue (2016, methodology): “To do so, we follow the approach of [16, 46] and extract an 8-dimensional vector of the location of the bounding box: xspatial = [xmin, ymin, xmax, ymax, xcenter, ycenter, wbox, hbox], where wbox and hbox denote the width and height of the bounding box, respectively.”
Modeling Relationships in Referential Expressions with Compositional Modular Networks (2016, methodology): “The problem of grounding referential expressions can be naturally formulated as a retrieval problem over image regions [19, 10, 24, 7, 32, 20].”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “, captioning an image or a video with natural language [9, 10, 39, 41, 47, 48, 42] or localizing desired objects within an image given a natural language description [31, 18, 49].”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, background): “Earlier works [16, 13, 45, 44, 42, 32] focused on identifying the referred object in an image.”