Grounding of Textual Phrases in Images by Reconstruction

By Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, Bernt Schiele. 2015

525 citations recorded by the source corpus.

Grounding (i.e. localizing) arbitrary, free-form textual phrases in visual content is a challenging problem with many applications for human-computer interaction and image-text reference resolution. Few datasets provide the ground truth spatial localization of phrases, thus it is desirable to learn from data with no or little grounding supervision. We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly. During training our approach encodes the phrase using a recurrent network language model and then learns to attend to the relevant image region in order to reconstruct the input phrase. At test time, the correct attention, i.e., the grounding, is evaluated. If grounding supervision is available it can be directly applied via a loss over the attention mechanism. We demonstrate the effectiveness of our approach on the Flickr 30k Entities and ReferItGame datasets with different levels of supervision, ranging from no supervision over partial supervision to full supervision. Our supervised variant improves by a large margin over the state-of-the-art on both datasets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “[72] combine CNNs with LSTMs for visual grounding.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “An orthogonal line of work has been on referring expressions: asking to what image region a natural language sentence refers to [60, 52, 65, 87, 88, 59, 36, 33].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “ase types, combined with the rarity of other phrase types, makes AP too unreliable. Thus, consistent with other works that perform evaluation on Flickr30K Entities (Fukui et al. 2016; Hu et al. 2016; Rohrbach et al. 2016; Wang et al. 2016b), we only report recall in this paper. 4.1.3 Phrase Localiza”
Neural Module Networks (2015, background): “While all the applications considered in this paper involve visual question answering, the architecture is much more general, and might easily be applied to visual referring expression resolution [9, 34] or question answering about natural language texts [15].”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, methodology): “[38] get higher results, with a R@1 of over 47%, but they use a much more complex method that includes LSTMs with a phrase reconstruction objective.”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, methodology): “We base our grounding approach on the fully-supervised version of GroundeR (Rohrbach et al., 2016).”
TVQA: Localized, Compositional Video Question Answering (2018, background): “For image-based object grounding, there has been much work on phrase grounding (Plummer et al., 2015; Wang et al., 2016b; Rohrbach et al., 2016) and referring expression comprehension (Hu et al., 2016; Yu et al., 2016; Nagaraja et al., 2016; Yu et al., 2017, 2018b).”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “Other recent work uses joint embedding model [21, 25, 15, 4] to compute P (o|r) directly.”
Localizing Moments in Video with Natural Language (2017, background): “[27, 29, 40] tackle the task of object grounding in which sentence fragments in a description are localized to speciﬁc image regions.”
Natural Language Object Retrieval (2015, methodology): “Concurrent with this paper, [27] uses an attention model to ground referential phrases in image descriptions by attending to regions where the phrases can be best reconstructed.”
Modeling Context in Referring Expressions (2016, background): “Three recent approaches for referring expression generation [26] and comprehension [14,33] also take a deep learning approach.”
Neural Baby Talk (2018, background): “Another related line of work is on resolving referring expressions [21] (or description-based object retrieval [36, 17, 18, 39] – given a description of an object in the image, identify which object is being referred to) or referring expression generation [21, 29, 31, 50] (given an object in the ima”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Existing work in visual grounding [38, 30, 25, 40, 35] has addressed the problem of passively perceiving a static image and mapping a referential expression to a bounding box [38, 30, 25] or a segmentation mask [24, 27, 55], exploring various techniques including proposal generation [10] and relatio”
Modeling Relationships in Referential Expressions with Compositional Modular Networks (2016, methodology): “The problem of grounding referential expressions can be naturally formulated as a retrieval problem over image regions [19, 10, 24, 7, 32, 20].”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, methodology): “Like [52], MCB uses softmax to map a phrase to a the single best region proposal from the image, with all the other regions (including ones having a high overlap with the ground truth) designated as negatives.”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, background): “Earlier works [16, 13, 45, 44, 42, 32] focused on identifying the referred object in an image.”
Weakly-Supervised Learning of Visual Relations (2017, background): “With the appearance of new datasets providing box-level natural language annotations [26, 27, 32, 37], recent works have also investigated caption generation at the level of image regions for the tasks of natural language object retrieval [19, 32, 41] or dense captioning [21].”