Image Description using Visual Dependency Representations

By Desmond Elliott, Frank Keller. 2013

309 citations recorded by the source corpus.

Describing the main event of an image involves identifying the objects depicted and predicting the relationships between them. Previous approaches have represented images as unstructured bags of regions, which makes it difficult to accurately predict meaningful relationships between regions. In this paper, we introduce visual dependency representations to capture the relationships between the objects in an image, and hypothesize that this representation can improve image description. We test this hypothesis using a new data set of region-annotated images, associated with visual dependency representations and gold-standard descriptions. We describe two template-based description generation models that operate over visual dependency representations. In an image description task, we find that these models outperform approaches that rely on object proximity or corpus information to generate descriptions on both automatic measures and on human judgements.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

VizWiz Grand Challenge: Answering Visual Questions from Blind People (2018, methodology): “We evaluate with respect to four metrics: Accuracy [8], CIDEr [42], BLEU4 [32], and METEOR [16].”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Show and tell: A neural image caption generator (2014, background): “…at time t via the three gates; the cell value is fed back via the forget gate; the predicted word at time t− 1 is fed back in addition to the memory output m at time t into the Softmax for word prediction. read its input (input gate i) and whether to output the new cell value (output gate o).”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
CIDEr: Consensus-based image description evaluation (2014, background): “Unfortunately, these metrics have been shown to correlate weakly with human judgment [22, 11, 4, 18].”
Movie Description (2016, methodology): “As known from literature (Chen et al. 2015; Elliott and Keller 2013; Vedantam et al. 2015), automatic evaluation measures do not always agree with the human evaluation.”
TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, background): “There has been a recent spike in efforts to automatically describe visual content in natural language (Yang et al., 2011; Kulkarni et al., 2011; Li et al., 2011; Farhadi et al., 2010; Krishnamoorthy et al., 2013; Elliott and Keller, 2013; Yu and Siskind, 2013; Socher et al., 2014).”
Comparing Automatic Evaluation Measures for Image Description (2014, methodology): “The test data of Elliott and Keller (2013) contains 101 images paired with three reference descriptions. The images were taken from the PASCAL VOC Action Recognition Task, the reference descriptions were collected from Mechanical Turk, and the judgements were also collected from Mechanical Turk. Ell”
Image Captioning with Semantic Attention (2016, methodology): “The topdown paradigm [4, 35, 26, 16, 8, 37, 25] starts from a “gist” of an image and converts it into words, while the bottom-up one [12, 19, 23, 9, 20, 11, 22] first comes up with words describing various aspects of an image and then combines them.”
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (2016, methodology): “An “encoder” RNN reads the source sentence and transforms it into a rich ﬁxed-length vector representation, which in turn in used as the initial hidden state of a “decoder” RNN that generates the target sentence.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “…the system’s rendition of the content (e.g. Lester & Porter, 1997; Sripada et al., 2005; Hunter et al., 2012), a criterion often used in subjective evaluations of image-captioning systems as well (e.g. Kulkarni et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Elliott & Keller, 2013).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “mmatically correct captions. However, templates are predefined and cannot generate variable-length captions. Moreover, later on, parsing based language models have been introduced in image captioning [2, 32, 76, 77, 101] which are more powerful than fixed template-based methods. Therefore, in this p”
See No Evil, Say No Evil: Description Generation from Densely Labeled Images (2014, methodology): “…et al., 2011), using text surrounding an image in a news article (Feng and Lapata, 2010), ﬁlling visual sentence templates (Kulkarni et al., 2011; Yang et al., 2011; Elliott and Keller, 2013), and stitching together existing sentence descriptions (Gupta and Mannem, 2012; Kuznetsova et al., 2012).”
Captioning Images Taken by People Who Are Blind (2020, methodology): “e trying to learn about. Image Captioning Datasets. Over the past decade, nearly 20 publicly-shared captioning datasets have been created to support the development of automated captioning algorithms [6,14,15,17,18,20,24,25,26,29,30,40, 43,50,51,54]. The recent trend has been to include a larger num”
Visual Storytelling (2016, background): “…captioning (Lin et al., 2014; Karpathy and Fei-Fei, 2015; Vinyals et al., 2015; Xu et al., 2015; Chen et al., 2015; Young et al., 2014; Elliott and Keller, 2013), question answering (Antol et al., 2015; Ren et al., 2015; Gao et al., 2015; Malinowski and Fritz, 2014), visual phrases (Sadeghi and…”
Visual to Text: Survey of Image and Video Captioning (2019, background): “(3) METEOR is the harmonic mean of unigram precision and recall, which is suitable for exact and paraphrase matchings between the reference X and candidate Y .”
Multimodal Machine Learning: A Survey and Taxonomy (2017, background): “[53] propose to explicitly model proximity relationships of objects for image description generation.”