I2T: Image Parsing to Text Description

By Benjamin Z. Yao, Xiong Yang, Liang Lin, Mun Wai Lee, Song-Chun Zhu. 2010

345 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Some approaches focus on the direction of describing the contents of images, formulated either as a task of mapping images to a fixed set of sentences written by people [5, 6], or as a task of automatically generating novel captions [7, 8, 9, 10, 11, 12].”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “A small child sits on a cement wall near white flower.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Early methods [19,31,37,65,79] focus on filling templates with extracted objects, attributes, and relationships.”
Attention on Attention for Image Captioning (2019, methodology): “Earlier approaches to image captioning are rule/template-based [48, 34] which generate slotted caption templates and use the outputs of object detection [30, 39, 38], attribute prediction and scene recognition to fill in the slots.”
Show and tell: A neural image caption generator (2014, background): “The choice of f in (3) is governed by its ability to deal with vanishing and exploding gradients [10], the most common challenge in designing and training RNNs.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “…Donahue et al. 2015; Fang et al. 2015; Farhadi et al. 2010; Hodosh et al. 2013; Karpathy and Fei-Fei 2015; Kiros et al. 2014; Klein et al. 2014; Lev et al. 2015; Kulkarni et al. 2011; Lebret et al. 2015; Ma et al. 2015; Mao et al. 2015b; Ordonez et al. 2011; Vinyals et al. 2015; Yao et al. 2010).”
ReferItGame: Referring to Objects in Photographs of Natural Scenes (2014, methodology): “Bottom up methods (Kulkarni et al., 2011; Yang et al., 2011; Yao et al., 2010) rely on visual classi-ﬁers to ﬁrst recognize image content and then construct captions from scratch, perhaps with some Figure 1: An example game.”
From captions to visual concepts and back (2014, neutral): “Previous work utilizes syntactic and semantic constraints in the generation process [28, 45, 23, 21, 20, 44], and we compare against prior state of the art in this line of work.”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, background): “Predicting natural language desriptions of still images has received considerable attention, with some of the earliest works by Aker and Gaizauskas (2010), Farhadi et al. (2010), Yao et al. (2010), and Kulkarni et al. (2011) amongst others.”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, background): “Numerous papers have explored the area of generating novel image descriptions [7, 36, 19, 37, 28, 11, 20, 17].”
TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, background): “The first focuses on describing exactly those items (e.g., objects, attributes) that are detected by vision recognition, which subsequently confines what should be described and how (Yao et al., 2010; Kulkarni et al., 2011; Kojima et al., 2002).”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, methodology): “Image and video description has been tack-led by using established algorithms [42, 10, 25, 17, 32].”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “While early approaches were based on caption templates filled by using the output of pre-trained object detectors [14], [15], almost all recent captioning models integrate recurrent neural networks as language models [6], 2020 IEEE International Conference on Robotics and Automation (ICRA) 31 May - ”
Joint Video and Text Parsing for Understanding Events and Answering Queries (2013, methodology): “We extend the text generator described in Yao et al. [65] to generate text descriptions from our joint parse graphs.”
Phrase-based Image Captioning (2015, background): “Many of them use syntactic and semantic constraints in the generation process (Yao et al., 2010; Mitchell et al., 2012; Kulkarni et al., 2011; Kuznetsova et al., 2012).”
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (2016, methodology): “An “encoder” RNN reads the source sentence and transforms it into a rich ﬁxed-length vector representation, which in turn in used as the initial hidden state of a “decoder” RNN that generates the target sentence.”
Video In Sentences Out (2012, background): “There has also been work on generating sentential descriptions of static images (Farhadi et al., 2009; Kulkarni et al., 2011; Yao et al., 2010).”
Choosing Linguistics over Vision to Describe Images (2012, neutral): “For sentence generation, either templates (Kulkarni et al. 2011; Yang et al. 2011; Yao et al. 2008) are used, or complete sentences from the available descriptions (Farhadi et al.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “Yao et al. (2010) use semi-supervised methods to parse images into graphs and then generate text via a simple grammar.”