Choosing Linguistics over Vision to Describe Images

By Ankush Gupta, C. V. Jawahar. 2012

139 citations recorded by the source corpus.

In this paper, we address the problem of automatically generating human-like descriptions for unseen images, given a collection of images and their corresponding human-generated descriptions. Previous attempts for this task mostly rely on visual clues and corpus statistics, but do not take much advantage of the semantic information inherent in the available image descriptions. Here, we present a generic method which benefits from all these three sources (i.e. visual clues, corpus statistics and available descriptions) simultaneously, and is capable of constructing novel descriptions. Our approach works on syntactically and linguistically motivated phrases extracted from the human descriptions. Experimental evaluations demonstrate that our formulation mostly generates lucid and semantically correct descriptions, and significantly outperforms the previous methods on automatic evaluation metrics. One of the significant advantages of our approach is that we can generate multiple interesting descriptions for an image. Unlike any previous work, we also test the applicability of our method on a large dataset containing complex images with rich descriptions.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, background): “…be of great interest for people who aim to build systems that automatically associate image with sentences that describe them (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Gupta et al., 2012; Hodosh et al., 2013).”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, background): “…of Farhadi et al. (2010), who incorporate Lin's similarity into their model, but do not evaluate what benefit they obtain from it, WordNet's hypernym-hyponym relations have only been used superficially for associating images and text (Weston et al., 2010; Ordonez et al., 2011; Gupta et al., 2012).”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
CIDEr: Consensus-based image description evaluation (2014, methodology): “Most of the approaches use the UIUC Pascal Sentence [14, 22, 29, 37, 17] and the MS COCO datasets [19, 6, 10, 43, 27, 21] for evaluation.”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, neutral): “15 PPL B-1 B-2 B-3 B-4 LBL, Mnih & Hinton (2007) 9.29 0.321 0.145 0.064 - MLBLB-AlexNet, Kiros et al. (2014b) 9.86 0.393 0.211 0.112 - MLBLF-AlexNet, Kiros et al. (2014b) 9.90 0.387 0.209 0.115 - Gupta et al. (2012) - 0.15 0.06 0.01 - Gupta & Mannem (2012) - 0.33 0.18 0.07 - Ours-RNN-Base 7.”
Explain Images with Multimodal Recurrent Neural Networks (2014, neutral): “ BACK-OFF GT2 54.5 0.323 0.145 0.059 BACK-OFF GT3 55.6 0.312 0.131 0.059 LBL [26] 20.1 0.327 0.144 0.068 MLBL-B-DeCAF [16] 24.7 0.373 0.187 0.098 MLBL-F-DeCAF [16] 21.8 0.361 0.176 0.092 Gupta et al. [11] / 0.15 0.06 0.01 Gupta & Mannem [10] / 0.33 0.18 0.07 Ours-RNN-Base 7.77 0.3134 0.1168 0.08”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, background): “Numerous papers have explored the area of generating novel image descriptions [7, 36, 19, 37, 28, 11, 20, 17].”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Such systems either use a unimodal space to compare a query image to training images before caption retrieval (e.g. Ordonez et al., 2011; Gupta et al., 2012), or exploit a multimodal space representing proximity between images and captions (e.”
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016, methodology): “The method of Gupta et al. (2012) is another phrase-based approach.”
From Show to Tell: A Survey on Image Captioning (2021, background): “Excluding the earliest image captioning works [2], [3], [4], [5], [6], [7], [8], [9], [10], we focus on deep learning-based solutions.”
Im2Text and Text2Im: Associating Images and Texts for Cross-Modal Retrieval (2014, methodology): “In contrast to several popular methods such as [5, 6, 10, 11, 15, 19, 22, 35, 37] that assume presence of data from both the modalities (visual and textual) during the testing phase, our approach has a motivation similar to the few known works (e.”
Seeing What You're Told: Sentence-Guided Activity Recognition in Video (2013, background): “Much of the prior work on generating sentences to describe images [4, 7, 8, 12, 13, 18] and video [1, 6, 9, 10, 16] uses special-purpose natural-language-generation methods.”
RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, methodology): “In traditional method stage, retrieval-based [8, 27, 10] and template-based [19, 26, 36] methods are two common types of implementation for image captioning.”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Extended Abstract) (2015, background): “Much of the related work has focused on models that produce novel captions for images, e.g. [Kulkarni et al., 2011; Gupta et al., 2012; Kuznetsova et al., 2012; Mitchell et al., 2012].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “entailed description retrieval [2], [3], [4], [5], [6], [7] or template filling and hand-crafted natural language generation techniques [8], [9], [10], [11], [12], [13], [14], [15].”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “Example-based Retrieval Image captioning ⇒ [58], [162] Media retrieval ⇔ [199], [239] Visual speech ⇒ [27] Image captioning ⇔ [102], [103] Combination Image captioning ⇒ [77], [119], [124]”
Top-down and bottom-up cues for scene text recognition (2012, background): “This could help in building vision systems, which can solve higher level semantic tasks, such as [15, 19].”
Deep correlation for matching images and text (2015, methodology): “from) natural language [5, 6, 11, 15, 16, 23, 25, 26, 29, 34, 35, 36, 42, 52] is one of the ultimate goals of computer vision (CV), natural language processing (NLP) and machine learning (ML).”
Open-book Video Captioning with Retrieve-Copy-Generate Network (2021, methodology): “Reviewing the early retrieval-based visual description methods can be roughly divided into two categories: [21, 9, 16] cast the problem as retrieval from a visual space problem and transfer the retrieved descriptions to a novel image/video; [5, 25, 11] cast the problem as retrieval from a multimodal”