What Are You Talking About? Text-to-Image Coreference

By Chen Kong, Dahua Lin, Mohit Bansal, Raquel Urtasun, Sanja Fidler. 2014

223 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “The field has tackled a diverse set of tasks such as captioning [15, 13, 40, 39, 24], alignment [13, 17, 38], Q&A [22, 21], visual model learning from textual descriptions [9, 29], and semantic visual search with natural multisentence queries [19].”
Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [10, 11, 20, 54], video/movie description [43, 51, 52], text-to-image coreference/grounding [16, 22, 37, 39, 42], Visual Madlibs [60], and of course, visual question answering (VQA) [4,”
VQA: Visual Question Answering (2015, background): “Several recent papers have explored tasks at the intersection of vision and language that are easier to evaluate than image captioning, such as coreference resolution [22, 36] or generating referring expressions [20, 35] for a particular object in an image that”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “urpose of natural language description. Finally, there exist a few specialized datasets with extensive annotations, but more limited domains of applicability than Flickr30k Entities or Visual Genome. Kong et al. (2014) have taken the 1,449 RGB-D images of static indoor scenes from the NYUv2 dataset ”
Neural Module Networks (2015, background): “This grounding task has previously been approached in [18, 32, 17, 20, 14], where the authors tried to localize phrases in an image.”
Visual7W: Grounded Question Answering in Images (2015, background): “The object-level groundings enable examining the object distributions and resolve the coreference ambiguity [19, 35].”
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input (2014, background): “Although others [7, 8] propose interesting alternatives for learning the language binding, it is unclear if such approaches can be used to provide answers on questions.”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, background): “Most available datasets focus on image [17, 22, 46, 50] or video description [5, 32, 9].”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “[19] is one of the few works which explicitly addresses language grounding in images.”
Natural Language Object Retrieval (2015, methodology): “[21] uses a structure prediction model to align text to image and reasons about object co-reference in text for 3D scene parsing.”
Unsupervised Semantic Parsing of Video Collections (2015, background): “Similarly, Zitnick et al.[61, 60] used abstracted clip-arts to understand spatial relations of objects and their language correspondences.”
Towards a Visual Turing Challenge (2014, background): “The p rogress in both fields has inspired researchers to build holistic architectures for challengi ng grounding [14, 15], natural language generation from image/video [16, 17, 18], image-to-sentence a lignment [19, 20, 21, 22], and recently presented question-answering problems [23, 24, 25, 26, 27]”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Structured Models Structured models in visual understanding have been explored for language grounding, where language determines the graph structures involved in prediction [29, 21, 37, 16].”
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences (2015, background): “ots and autonomous agents to follow free-form navigational route instructions (MacMahon, Stankiewicz, and Kuipers 2006; Kollar et al. 2010; Chen and Mooney 2011; Chen 2012; Kim and Mooney 2012; 2013; Kong et al. 2014; Hemachandra et al. 2015). These methods solve what Harnad (1990) refers to as the ”
Modeling Context Between Objects for Referring Expression Understanding (2016, neutral): “[19] learn visual grounding for nouns in descriptions of indoor scenes in a supervised manner.”
Captioning Images Taken by People Who Are Blind (2020, methodology): “e trying to learn about. Image Captioning Datasets. Over the past decade, nearly 20 publicly-shared captioning datasets have been created to support the development of automated captioning algorithms [6,14,15,17,18,20,24,25,26,29,30,40, 43,50,51,54]. The recent trend has been to include a larger num”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “Many works have tackled this problem [37,38,39,40], enforcing that language terms be grounded in the image.”
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering (2016, background): “…techniques (Krishnamurthy and Kollar, 2013; Matuszek et al, 2012), there is a recent trend of machine learning-based approaches (Karpathy and Fei-Fei, 2015; Karpathy et al, 2014; Akata et al, 2016; Kong et al, 2014; Hu et al, 2016; Rohrbach et al, 2015a; Mao et al, 2016) to find the associations.”