Learning the Visual Interpretation of Sentences

By C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende. 2013

160 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “[14] use a Conditional Random Field to reason about spatial relationships in cartoon scenes and their relation to natural language descriptions.”
VQA: Visual Question Answering (2015, background): “To attract researchers interested in exploring the high-level reasoning required for VQA, but not the low-level vision tasks, we create a new abstract scenes dataset [1, 48, 49, 50] containing 50K scenes.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Visual7W: Grounded Question Answering in Images (2015, background): “There have been years of effort in connecting the visual and textual information for joint learning [2, 19, 33, 35, 40, 52].”
Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images (2015, positive): “Combining RNNs and CNNs for description of visual content The task of describing visual content like still images as well as videos has been successfully addressed with a combination of the previous two ideas [28, 23, 3, 22, 8].”
Yin and Yang: Balancing and Answering Binary Visual Questions (2015, methodology): “To determine which objects are being referred to by the P and S arguments, we follow the idea in [43] and Figure 3: Most plausible words for an object determined using mutual information.”
ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering (2015, methodology): “Current state-of-the-art methods in VQA [7, 18, 21] and image captioning [19, 6, 10, 27, 28, 31, 25] generally apply a CNN to extract visual features and an LSTM model as a decoder to generate answers or captions.”
Scene Graph Generation by Iterative Message Passing (2017, background): “One of the most popular ways of representing a visual scene is through text descriptions [14, 34, 44].”
Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation (2014, background): “Other works [49, 9, 42] have focused on developing bi-directional mappings.”
Unsupervised Semantic Parsing of Video Collections (2015, methodology): “[63, 62] used abstracted clip-arts to understand spatial relations of objects and their language correspondences.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Datasets typically consist of images paired with one or more human-authored captions (mostly in English) and vary from artificially created scenes (Zitnick et al., 2013) to real photographs.”
Visual Relationship Detection with Language Priors (2016, background): “Some papers explicitly collected relationships in images [25,26,27,28,29] and videos [27,30,31] and helped models map these relationships from images to language.”
Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasks (2015, methodology): “Approaches that automatically produce a textual description of images [22, 14, 29] or synthesize scenes corresponding to input textual descriptions [52] can benefit from reasoning about all these different “W” questions and other high-level information.”
Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval (2015, methodology): “For example, we can reuse a scene graph parser for systems that generate 2D-scenes (Zitnick et al., 2013) or 3Dscenes (Chang et al.”
Captioning Images Taken by People Who Are Blind (2020, methodology): “e trying to learn about. Image Captioning Datasets. Over the past decade, nearly 20 publicly-shared captioning datasets have been created to support the development of automated captioning algorithms [6,14,15,17,18,20,24,25,26,29,30,40, 43,50,51,54]. The recent trend has been to include a larger num”
Inferring the Why in Images (2014, background): “Language in Vision: The community has been incorporating natural language into computer vision over the last decade to great success, from generating sentences from images [19], producing visual models from sentences [36, 33], and aiding in contextual models [26, 21] to name a few.”
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (2016, methodology): “In this work we investigate a Bag-Of-Words (BOW), a CNN, and two RNN variants (LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Cho et al, 2014)) to encode the question.”