Exploring Nearest Neighbor Approaches for Image Captioning

By Jacob Devlin, Saurabh Gupta, Ross B. Girshick, Margaret Mitchell, C. Lawrence Zitnick. 2015

201 citations recorded by the source corpus.

We explore a variety of nearest neighbor baseline approaches for image captioning. These approaches find a set of nearest neighbor images in the training set from which a caption may be borrowed for the query image. We select a caption for the query image by finding the caption that best represents the "consensus" of the set of candidate captions gathered from the nearest neighbor images. When measured by automatic evaluation metrics on the MS COCO caption evaluation server, these approaches perform as well as many recent approaches that generate novel captions. However, human studies show that a method that generates novel captions is still preferred over the nearest neighbor approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “For instance, in image captioning, a simple baseline of utilizing the captions of nearest neighbors in the training set was shown to yield impressive BLEU scores (Devlin et al., 2015).”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “For example, it has been pointed out that language can inadvertently impose strong priors that result in seemingly impressive performance, without any understanding of the visual content in the underlying model [15].”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “It is worth noting that a more sophisticated nearest neighbor approach based on multiple neighbors and a consensus voting strategy can be very competitive with state of the art CNN-RNN approaches similar to the one explored in this work [67].”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “lso provides a strong prior that can result in good superﬁcial performance, without the underlying models truly understanding the visual content. This phenomenon has been observed in image captioning [6] as well as visual question answering [47,49,16,18, 1]. For instance, in the VQA [3] dataset, the”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “Rather than searching for predefined labels, which can introduce search engine bias [76, 16, 20], we collect images from movie scenes.”
Exploring Models and Data for Image Question Answering (2015, background): “[30] showed that a nearest neighbors baseline approach actually performs very well.”
Yin and Yang: Balancing and Answering Binary Visual Questions (2015, background): “borrowed from nearest neighbor images result in a surprisingly high performance [10].”
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding (2016, methodology): “[41] we simply use a 1Nearest Neighbor baseline computed using AlexNet fc7 outputs averaged over frames, and use the caption from that nearest neighbor in the training set.”
Simple Baseline for Visual Question Answering (2015, background): “Combining Natural Language Processing with Computer Vision for high-level scene interpretation is a recent trend, e.g., image captioning [10, 15, 7, 4].”
Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions? (2016, background): “Unlike image captioning, where a coarse understanding of an image is often sufficient for producing generic descriptions (Devlin et al., 2015), visual questions selectively target different areas of an image including background details and underlying context.”
Image Captioning with Semantic Attention (2016, background): “It has been shown that these weakly annotated images can be exploited to learn visual concepts [38], text-image embedding [14] and image captions [7].”
In Defense of Grid Features for Visual Question Answering (2020, background): “For example, deep learning features led to remarkable improvements in image captioning [9, 44, 8].”
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (2016, methodology): “An “encoder” RNN reads the source sentence and transforms it into a rich ﬁxed-length vector representation, which in turn in used as the initial hidden state of a “decoder” RNN that generates the target sentence.”
Guided Open Vocabulary Image Captioning with Constrained Beam Search (2016, methodology): “Recently, models incorporating recurrent neural networks (RNNs) have demonstrated promising re-sults on this challenging task (Vinyals et al., 2015; Fang et al., 2015; Devlin et al., 2015), leveraging new benchmark datasets such as the MSCOCO dataset (Lin et al., 2014).”